Package org.apache.drill.exec.physical.impl.scan.v3.lifecycle

Class ScanLifecycle

java.lang.Object
org.apache.drill.exec.physical.impl.scan.v3.lifecycle.ScanLifecycle
Direct Known Subclasses:
FileScanLifecycle
public class ScanLifecycle extends Object
/** Basic scan framework for a set of "managed" readers and which uses the scan schema tracker to evolve the scan output schema. Readers are created and managed via a reader factory class unique to each type of scan. The reader factory also provides the scan-specific schema negotiator to be passed to the reader.

Lifecycle

The options provided in the ScanLifecycleBuilder are sufficient to drive the entire scan operator functionality. Schema resolution and projection is done generically and is the same for all data sources. Only the reader (created via the factory class) differs from one type of file to another.

The framework achieves the work described below by composing a set of detailed classes, each of which performs some specific task. This structure leaves the reader to simply infer schema and read data.

Reader Integration

The details of how a file is structured, how a schema is inferred, how data is decoded: all that is encapsulated in the reader. The only real Interaction between the reader and the framework is:
  • The reader factory creates a reader and the corresponding schema negotiator.
  • The reader "negotiates" a schema with the framework. The framework knows the projection list from the query plan, knows something about data types (whether a column should be scalar, a map or an array), and knows about the schema already defined by prior readers. The reader knows what schema it can produce (if "early schema.") The schema negotiator class handles this task.
  • The reader reads data from the file and populates value vectors a batch at a time. The framework creates the result set loader to use for this work. The schema negotiator returns that loader to the reader, which uses it during read.

    A reader may be "late schema", true "schema on read." In this case, the reader simply tells the result set loader to create a new column reader on the fly. The framework will work out if that new column is to be projected and will return either a real column writer (projected column) or a dummy column writer (unprojected column.)

  • The reader then reads batches of data until all data is read. The result set loader signals when a batch is full; the reader should not worry about this detail itself.
  • The reader then releases its resources.

See ScanSchemaTracker for details about how the scan schema evolves over the scan lifecycle.

Lifecycle

Coordinates the components that make up a scan implementation:
  • ScanSchemaTracker which resolves the scan schema over the lifetime of the scan.
  • Implicit columns manager which identifies and populates implicit file columns, partition columns, and Drill's internal metadata columns.
  • The actual readers which load (possibly a subset of) the columns requested from the input source.

Implicit columns are unique to each storage plugin. At present, they are defined only for the file system plugin. To handle such variation, each extension defines a subclass of the ScanLifecycleBuilder class to create the implicit columns manager (and schema negotiator) unique to a certain kind of scan.

Each reader is tracked by a ReaderLifecycle which handles:

  • Setting up the ResultSetLoader for the reader.
  • The concrete values for implicit columns for that reader (and its file, if file-based.)
  • The missing columns handler which "makes up" values for projected columns not read by the reader.
  • Batch asssembler, which combines the three sources of vectors to create the output batch with the schema specified by the schema tracker.

Publishing the Final Result Set

This class "publishes" a vector container that has the final, projected form of a scan. The projected schema include:
  • Columns from the reader.
  • Static columns, such as implicit or partition columns.
  • Null columns for items in the select list, but not found in either of the above two categories.
The order of columns is that set by the select list (or, by the reader for a SELECT * query.

See Also: