Interface SchemaNegotiator
- All Known Subinterfaces:
FileSchemaNegotiator
- All Known Implementing Classes:
FileSchemaNegotiatorImpl,SchemaNegotiatorImpl
Regardless of the schema type, the result of building the schema is a result set loader used to prepare batches for use in the query. The reader can simply read all columns, allowing the framework to discard unwanted values. Or for efficiency, the reader can check the column metadata to determine if a column is projected, and if not, then don't even read the column from the input source.
Defined Schema
If defined, the execution plan provides the output schema (presumably computed from an accurate metadata source.) The reader must populate the proscribed rows, performing column type conversions as needed. The reader can determine if the schema is defined by callinghasOutputSchema().
At present, the scan framework filters the "provided schema" against the project list so that this class presents only the actual output schema. Future versions may do the filtering in the planner, but the result for readers will be the same either way.
Dynamic Schema
A dynamic schema occurs when the plan does not specify a schema. Drill is unique in its support for "schema on read" in the sense that Drill does not know the schema until the reader defines it at scan time.The reader and scan framework coordinate to form the output schema. The reader offers the columns it has available. The scan framework uses the projection list to decide which to accept. Either way the scan framework provides a column reader for the column (returning a do-nothing "dummy" reader if the column is unprojected.)
With a dynamic schema, readers offer a schema in one of two ways:
The reader provides the table schema in one of two ways: early schema or late schema. Either way, the project list from the physical plan determines which table columns are materialized and which are not. Readers are provided for all table columns for readers that must read sequentially, but only the materialized columns are written to value vectors.
Early Dynamic Schema
Some readers can determine the source schema at the start of a scan. For example, a CSV file has headers, a Parquet file has footers, both of which define a schema. This case is called "early schema." The reader fefines the schema by callingtableSchema(TupleMetadata) to provide the known schema.
Late Dynamic Schema
Other readers don't know the input schema until the reader actually reads the data. For example, JSON typically has no schema, but does have sufficient structure (name/value pairs) to infer one.
The late schema reader calls RowSetLoader#addColumn() to
add each column as it is discovered during the scan.
Note that, to avoid schema conflicts, a late schema reader must define the full set of columns in the first batch, and must stick to that schema for all subsequent batches. This allows the reader to look one batch ahead to learn the columns.
Drill, however, cannot predict the future. Without a defined schema, downstream operators cannot know which columns might appear later in the scan, with which types. Today this is a strong guideline. Future versions may enforce this rule.
-
Method Summary
Modifier and TypeMethodDescriptionvoidbatchSize(int maxRecordsPerBatch) Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)build()Build the schema, plan the required projections and static columns and return a loader used to populate value vectors.context()Returns the error context to use for this reader: either the parent or the reader-specific context set insetErrorContext(CustomErrorContext).Returns the reader input schema: the schema which describes the set of columns this reader should produce.booleanReport whether the projection list is empty, as occurs in two cases: SELECT COUNT(*) ... -- empty project. SELECT a, b FROM table(c d) -- disjoint project.The context to use as a parent when creating a custom context.projectionFor(String colName) Returns the provided schema, if defined.voidschemaIsComplete(boolean isComplete) voidsetErrorContext(CustomErrorContext context) Specify an advanced error context which allows the reader to fill in custom context values.voidtableSchema(TupleMetadata schema) voidtableSchema(TupleMetadata schema, boolean isComplete) Specify the table schema if this is an early-schema reader.userName()Name of the user running the query.
-
Method Details
-
context
OperatorContext context() -
parentErrorContext
CustomErrorContext parentErrorContext()The context to use as a parent when creating a custom context.(Obtain the error context for this reader from the
ResultSetLoader. -
setErrorContext
Specify an advanced error context which allows the reader to fill in custom context values. -
errorContext
CustomErrorContext errorContext()Returns the error context to use for this reader: either the parent or the reader-specific context set insetErrorContext(CustomErrorContext). -
userName
String userName()Name of the user running the query. -
isProjectionEmpty
boolean isProjectionEmpty()Report whether the projection list is empty, as occurs in two cases:- SELECT COUNT(*) ... -- empty project.
- SELECT a, b FROM table(c d) -- disjoint project.
- Returns:
- true if no columns are projected, and the client can
make use of
ResultSetLoader.skipRows(int)to indicate the row count, false if at least one column is projected and so data must be written using the loader
-
projectionFor
-
providedSchema
TupleMetadata providedSchema()Returns the provided schema, if defined. The provided schema is a description of the source schema viewed as a Drill schema.If a schema is provided, the reader should use that schema, converting or ignoring columns as needed. A scan without a provided schema has a "dynamic" schema to be defined by the scan operator itself along with the column projection list.
The provided schema describes the columns that the reader is capable of providing. Any given query may choose to project a subset of these columns.
- Returns:
- the provided schema if the execution plan defines the output
schema,
nullif the schema should be computed dynamically from the source schema and column projections
-
inputSchema
TupleMetadata inputSchema()Returns the reader input schema: the schema which describes the set of columns this reader should produce. The schema can be fully dynamic (a wildcard), fully concrete (a list of columns and their types) or partially dynamic (a list of columns, but some types are not known.) If the reader has a choice of ways to map input columns to types, the reader must choose the type specified in the schema. Presumably, the specified types are those that the reader is capable of providing; there is no expectation that the reader will convert to arbitrary types.If the type is
LATE(dynamic), then the reader can choose the type. Subsequent readers in the same scan will then see the same column as concrete, with the type selected by the present reader.The reader is free to add additional columns. The internal mechanism will just ignore those columns and return a dummy reader for them.
The reader is also free to ignore columns. In this case, the internal mechanism will fill in
NULLor default values. -
tableSchema
Specify the table schema if this is an early-schema reader. Need not be called for a late-schema readers. The schema provided here, if any, is a base schema: the reader is free to discover additional columns during the read.- Parameters:
schema- the table schema if known at open timeisComplete- true if the schema is complete: if it can be used to define an empty schema-only batch for the first reader. Set to false if the schema is partial: if the reader must read rows to determine the full schema
-
tableSchema
-
schemaIsComplete
void schemaIsComplete(boolean isComplete) -
batchSize
void batchSize(int maxRecordsPerBatch) Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)- Parameters:
maxRecordsPerBatch- preferred number of record per batch
-
build
ResultSetLoader build()Build the schema, plan the required projections and static columns and return a loader used to populate value vectors. If the select list includes a subset of table columns, then the loader will be set up in table schema order, but the unneeded column loaders will be null, meaning that the batch reader should skip setting those columns.- Returns:
- the loader for the table with columns arranged in table schema order
-