Class ScanLifecycle
- Direct Known Subclasses:
FileScanLifecycle
Lifecycle
The options provided in theScanLifecycleBuilder
are
sufficient to drive the entire scan operator functionality.
Schema resolution and projection is done generically and is the same for all
data sources. Only the
reader (created via the factory class) differs from one type of file to
another.
The framework achieves the work described below by composing a set of detailed classes, each of which performs some specific task. This structure leaves the reader to simply infer schema and read data.
Reader Integration
The details of how a file is structured, how a schema is inferred, how data is decoded: all that is encapsulated in the reader. The only real Interaction between the reader and the framework is:- The reader factory creates a reader and the corresponding schema negotiator.
- The reader "negotiates" a schema with the framework. The framework knows the projection list from the query plan, knows something about data types (whether a column should be scalar, a map or an array), and knows about the schema already defined by prior readers. The reader knows what schema it can produce (if "early schema.") The schema negotiator class handles this task.
- The reader reads data from the file and populates value vectors a
batch at a time. The framework creates the result set loader to use for
this work. The schema negotiator returns that loader to the reader, which
uses it during read.
A reader may be "late schema", true "schema on read." In this case, the reader simply tells the result set loader to create a new column reader on the fly. The framework will work out if that new column is to be projected and will return either a real column writer (projected column) or a dummy column writer (unprojected column.)
- The reader then reads batches of data until all data is read. The result set loader signals when a batch is full; the reader should not worry about this detail itself.
- The reader then releases its resources.
See ScanSchemaTracker
for details about how the scan schema
evolves over the scan lifecycle.
Lifecycle
Coordinates the components that make up a scan implementation:ScanSchemaTracker
which resolves the scan schema over the lifetime of the scan.- Implicit columns manager which identifies and populates implicit file columns, partition columns, and Drill's internal metadata columns.
- The actual readers which load (possibly a subset of) the columns requested from the input source.
Implicit columns are unique to each storage plugin. At present, they
are defined only for the file system plugin. To handle such variation,
each extension defines a subclass of the ScanLifecycleBuilder
class to
create the implicit columns manager (and schema negotiator) unique to
a certain kind of scan.
Each reader is tracked by a ReaderLifecycle
which handles:
- Setting up the
ResultSetLoader
for the reader. - The concrete values for implicit columns for that reader (and its file, if file-based.)
- The missing columns handler which "makes up" values for projected columns not read by the reader.
- Batch asssembler, which combines the three sources of vectors to create the output batch with the schema specified by the schema tracker.
Publishing the Final Result Set
This class "publishes" a vector container that has the final, projected
form of a scan. The projected schema include:
- Columns from the reader.
- Static columns, such as implicit or partition columns.
- Null columns for items in the select list, but not found in either
of the above two categories.
The order of columns is that set by the select list (or, by the reader for
a SELECT * query.
- Columns from the reader.
- Static columns, such as implicit or partition columns.
- Null columns for items in the select list, but not found in either of the above two categories.
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionint
void
close()
context()
boolean
protected SchemaNegotiatorImpl
newNegotiator
(ReaderLifecycle readerLifecycle) options()
long
rowCount()
void
tallyBatch
(int rowCount)
-
Constructor Details
-
ScanLifecycle
-
-
Method Details
-
context
-
options
-
schemaTracker
-
vectorCache
-
readerFactory
-
hasOutputSchema
public boolean hasOutputSchema() -
errorContext
-
allocator
-
batchCount
public int batchCount() -
rowCount
public long rowCount() -
tallyBatch
public void tallyBatch(int rowCount) -
nextReader
-
newNegotiator
-
outputSchema
-
close
public void close()
-