org.apache.drill.exec.physical.impl.scan.v3

Interface SchemaNegotiator

All Known Subinterfaces:

FileSchemaNegotiator

All Known Implementing Classes:

FileSchemaNegotiatorImpl, SchemaNegotiatorImpl

public interface SchemaNegotiator

Negotiates the table schema with the scanner framework and provides context information for the reader. Scans use either a "dynamic" or a defined schema.

Regardless of the schema type, the result of building the schema is a result set loader used to prepare batches for use in the query. The reader can simply read all columns, allowing the framework to discard unwanted values. Or for efficiency, the reader can check the column metadata to determine if a column is projected, and if not, then don't even read the column from the input source.

Defined Schema

If defined, the execution plan provides the output schema (presumably computed from an accurate metadata source.) The reader must populate the proscribed rows, performing column type conversions as needed. The reader can determine if the schema is defined by calling hasOutputSchema().

At present, the scan framework filters the "provided schema" against the project list so that this class presents only the actual output schema. Future versions may do the filtering in the planner, but the result for readers will be the same either way.

Dynamic Schema

A dynamic schema occurs when the plan does not specify a schema. Drill is unique in its support for "schema on read" in the sense that Drill does not know the schema until the reader defines it at scan time.

The reader and scan framework coordinate to form the output schema. The reader offers the columns it has available. The scan framework uses the projection list to decide which to accept. Either way the scan framework provides a column reader for the column (returning a do-nothing "dummy" reader if the column is unprojected.)

With a dynamic schema, readers offer a schema in one of two ways:

The reader provides the table schema in one of two ways: early schema or late schema. Either way, the project list from the physical plan determines which table columns are materialized and which are not. Readers are provided for all table columns for readers that must read sequentially, but only the materialized columns are written to value vectors.

Early Dynamic Schema

Some readers can determine the source schema at the start of a scan. For example, a CSV file has headers, a Parquet file has footers, both of which define a schema. This case is called "early schema." The reader fefines the schema by calling tableSchema(TupleMetadata) to provide the known schema.

Late Dynamic Schema

Other readers don't know the input schema until the reader actually reads the data. For example, JSON typically has no schema, but does have sufficient structure (name/value pairs) to infer one.

The late schema reader calls RowSetLoader#addColumn() to add each column as it is discovered during the scan.

Note that, to avoid schema conflicts, a late schema reader must define the full set of columns in the first batch, and must stick to that schema for all subsequent batches. This allows the reader to look one batch ahead to learn the columns.

Drill, however, cannot predict the future. Without a defined schema, downstream operators cannot know which columns might appear later in the scan, with which types. Today this is a strong guideline. Future versions may enforce this rule.

Method Summary

Modifier and Type	Method and Description
void	batchSize(int maxRecordsPerBatch) Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)
ResultSetLoader	build() Build the schema, plan the required projections and static columns and return a loader used to populate value vectors.
OperatorContext	context()
CustomErrorContext	errorContext() Returns the error context to use for this reader: either the parent or the reader-specific context set in setErrorContext(CustomErrorContext).
TupleMetadata	inputSchema() Returns the reader input schema: the schema which describes the set of columns this reader should produce.
boolean	isProjectionEmpty() Report whether the projection list is empty, as occurs in two cases: SELECT COUNT(*) ... -- empty project. SELECT a, b FROM table(c d) -- disjoint project.
CustomErrorContext	parentErrorContext() The context to use as a parent when creating a custom context.
ProjectedColumn	projectionFor(String colName)
TupleMetadata	providedSchema() Returns the provided schema, if defined.
void	schemaIsComplete(boolean isComplete)
void	setErrorContext(CustomErrorContext context) Specify an advanced error context which allows the reader to fill in custom context values.
void	tableSchema(TupleMetadata schema)
void	tableSchema(TupleMetadata schema, boolean isComplete) Specify the table schema if this is an early-schema reader.
String	userName() Name of the user running the query.

Method Detail

context

OperatorContext context()

parentErrorContext

CustomErrorContext parentErrorContext()

The context to use as a parent when creating a custom context.

(Obtain the error context for this reader from the ResultSetLoader.

setErrorContext

void setErrorContext(CustomErrorContext context)

Specify an advanced error context which allows the reader to fill in custom context values.

errorContext

CustomErrorContext errorContext()

Returns the error context to use for this reader: either the parent or the reader-specific context set in setErrorContext(CustomErrorContext).

userName

String userName()

Name of the user running the query.

isProjectionEmpty

boolean isProjectionEmpty()

Report whether the projection list is empty, as occurs in two cases:

• SELECT COUNT(*) ... -- empty project.

SELECT a, b FROM table(c d) -- disjoint project.

Returns:

true if no columns are projected, and the client can make use of ResultSetLoader.skipRows(int) to indicate the row count, false if at least one column is projected and so data must be written using the loader

projectionFor

ProjectedColumn projectionFor(String colName)

providedSchema

TupleMetadata providedSchema()

Returns the provided schema, if defined. The provided schema is a description of the source schema viewed as a Drill schema.

If a schema is provided, the reader should use that schema, converting or ignoring columns as needed. A scan without a provided schema has a "dynamic" schema to be defined by the scan operator itself along with the column projection list.

The provided schema describes the columns that the reader is capable of providing. Any given query may choose to project a subset of these columns.

Returns:

the provided schema if the execution plan defines the output schema, null if the schema should be computed dynamically from the source schema and column projections

inputSchema

TupleMetadata inputSchema()

Returns the reader input schema: the schema which describes the set of columns this reader should produce. The schema can be fully dynamic (a wildcard), fully concrete (a list of columns and their types) or partially dynamic (a list of columns, but some types are not known.) If the reader has a choice of ways to map input columns to types, the reader must choose the type specified in the schema. Presumably, the specified types are those that the reader is capable of providing; there is no expectation that the reader will convert to arbitrary types.

If the type is LATE (dynamic), then the reader can choose the type. Subsequent readers in the same scan will then see the same column as concrete, with the type selected by the present reader.

The reader is free to add additional columns. The internal mechanism will just ignore those columns and return a dummy reader for them.

The reader is also free to ignore columns. In this case, the internal mechanism will fill in NULL or default values.

tableSchema

void tableSchema(TupleMetadata schema,
                 boolean isComplete)

Specify the table schema if this is an early-schema reader. Need not be called for a late-schema readers. The schema provided here, if any, is a base schema: the reader is free to discover additional columns during the read.

Parameters:

schema - the table schema if known at open time

isComplete - true if the schema is complete: if it can be used to define an empty schema-only batch for the first reader. Set to false if the schema is partial: if the reader must read rows to determine the full schema

tableSchema

void tableSchema(TupleMetadata schema)

schemaIsComplete

void schemaIsComplete(boolean isComplete)

batchSize

void batchSize(int maxRecordsPerBatch)

Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)

Parameters:

maxRecordsPerBatch - preferred number of record per batch

build

ResultSetLoader build()

Build the schema, plan the required projections and static columns and return a loader used to populate value vectors. If the select list includes a subset of table columns, then the loader will be set up in table schema order, but the unneeded column loaders will be null, meaning that the batch reader should skip setting those columns.

Returns:

the loader for the table with columns arranged in table schema order