org.apache.drill.exec.physical.impl.scan.framework

Interface SchemaNegotiator

All Known Subinterfaces:

ColumnsSchemaNegotiator, FileScanFramework.FileSchemaNegotiator

All Known Implementing Classes:

ColumnsScanFramework.ColumnsSchemaNegotiatorImpl, FileScanFramework.FileSchemaNegotiatorImpl, SchemaNegotiatorImpl

public interface SchemaNegotiator

Negotiates the table schema with the scanner framework and provides context information for the reader. Scans use either a "dynamic" or a defined schema.

Regardless of the schema type, the result of building the schema is a result set loader used to prepare batches for use in the query. The reader can simply read all columns, allowing the framework to discard unwanted values. Or for efficiency, the reader can check the column metadata to determine if a column is projected, and if not, then don't even read the column from the input source.

Defined Schema

If defined, the execution plan provides the output schema (presumably computed from an accurate metadata source.) The reader must populate the proscribed rows, performing column type conversions as needed. The reader can determine if the schema is defined by calling hasOutputSchema().

At present, the scan framework filters the "provided schema" against the project list so that this class presents only the actual output schema. Future versions may do the filtering in the planner, but the result for readers will be the same either way.

Dynamic Schema

A dynamic schema occurs when the plan does not specify a schema. Drill is unique in its support for "schema on read" in the sense that Drill does not know the schema until the reader defines it at scan time.

The reader and scan framework coordinate to form the output schema. The reader offers the columns it has available. The scan framework uses the projection list to decide which to accept. Either way the scan framework provides a column reader for the column (returning a do-nothing "dummy" reader if the column is unprojected.)

With a dynamic schema, readers offer a schema in one of two ways:

The reader provides the table schema in one of two ways: early schema or late schema. Either way, the project list from the physical plan determines which table columns are materialized and which are not. Readers are provided for all table columns for readers that must read sequentially, but only the materialized columns are written to value vectors.

Early Dynamic Schema

Some readers can determine the source schema at the start of a scan. For example, a CSV file has headers, a Parquet file has footers, both of which define a schema. This case is called "early schema." The reader fefines the schema by calling #tableSchema(TupleMetadata) to provide the known schema.

Late Dynamic Schema

Other readers don't know the input schema until the reader actually reads the data. For example, JSON typically has no schema, but does have sufficient structure (name/value pairs) to infer one.

The late schema reader calls RowSetLoader#addColumn() to add each column as it is discovered during the scan.

Note that, to avoid schema conflicts, a late schema reader must define the full set of columns in the first batch, and must stick to that schema for all subsequent batches. This allows the reader to look one batch ahead to learn the columns.

Drill, however, cannot predict the future. Without a defined schema, downstream operators cannot know which columns might appear later in the scan, with which types. Today this is a strong guideline. Future versions may enforce this rule.

Method Summary

Modifier and Type	Method and Description
void	batchSize(int maxRecordsPerBatch) Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)
ResultSetLoader	build() Build the schema, plan the required projections and static columns and return a loader used to populate value vectors.
OperatorContext	context()
com.typesafe.config.Config	drillConfig()
boolean	hasProvidedSchema() Report if the execution plan defines a provided schema.
boolean	isProjectionEmpty() Report whether the projection list is empty, as occurs in two cases: SELECT COUNT(*) ... -- empty project. SELECT a, b FROM table(c d) -- disjoint project.
void	limit(long limit) Push down a LIMIT into the scan.
CustomErrorContext	parentErrorContext() The context to use as a parent when creating a custom context.
TupleMetadata	providedSchema() Returns the provided schema, if defined.
OptionSet	queryOptions()
void	setErrorContext(CustomErrorContext context) Specify an advanced error context which allows the reader to fill in custom context values.
void	tableSchema(TupleMetadata schema, boolean isComplete) Specify the table schema if this is an early-schema reader.
String	userName() Name of the user running the query.

Method Detail

context

OperatorContext context()

drillConfig

com.typesafe.config.Config drillConfig()

queryOptions

OptionSet queryOptions()

setErrorContext

void setErrorContext(CustomErrorContext context)

Specify an advanced error context which allows the reader to fill in custom context values.

userName

String userName()

Name of the user running the query.

hasProvidedSchema

boolean hasProvidedSchema()

Report if the execution plan defines a provided schema. If so, the reader should use that schema, converting or ignoring columns as needed. A scan without a provided schema has a "dynamic" schema to be defined by the scan operator itself along with the column projection list.

Returns:

true if the execution plan defines the output schema, false if the schema should be computed dynamically from the source schema and column projections

providedSchema

TupleMetadata providedSchema()

Returns the provided schema, if defined. The provided schema is a description of the source schema viewed as a Drill schema.

Returns:

the output schema, if hasProvidedSchema() returns true, null otherwise

tableSchema

void tableSchema(TupleMetadata schema,
                 boolean isComplete)

Specify the table schema if this is an early-schema reader. Need not be called for a late-schema readers. The schema provided here, if any, is a base schema: the reader is free to discover additional columns during the read.

Should only be called if the schema is dynamic, that is, if hasProvidedSchema() returns false.

Parameters:

schema - the table schema if known at open time

isComplete - true if the schema is complete: if it can be used to define an empty schema-only batch for the first reader. Set to false if the schema is partial: if the reader must read rows to determine the full schema

batchSize

void batchSize(int maxRecordsPerBatch)

Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)

Parameters:

maxRecordsPerBatch - preferred number of record per batch

limit

void limit(long limit)

Push down a LIMIT into the scan. This is a per-reader limit, not an overall scan limit.

build

ResultSetLoader build()

Build the schema, plan the required projections and static columns and return a loader used to populate value vectors. If the select list includes a subset of table columns, then the loader will be set up in table schema order, but the unneeded column loaders will be null, meaning that the batch reader should skip setting those columns.

Returns:

the loader for the table with columns arranged in table schema order

isProjectionEmpty

boolean isProjectionEmpty()

Report whether the projection list is empty, as occurs in two cases:

• SELECT COUNT(*) ... -- empty project.

SELECT a, b FROM table(c d) -- disjoint project.

Returns:

true if no columns are projected, and the client can make use of ResultSetLoader.skipRows(int) to indicate the row count, false if at least one column is projected and so data must be written using the loader

parentErrorContext

CustomErrorContext parentErrorContext()

The context to use as a parent when creating a custom context.

(Obtain the error context for this reader from the ResultSetLoader.