Package org.apache.drill.exec.physical.impl.scan.v3

Interface SchemaNegotiator

All Known Subinterfaces:

FileSchemaNegotiator

All Known Implementing Classes:

FileSchemaNegotiatorImpl, SchemaNegotiatorImpl

public interface SchemaNegotiator

Negotiates the table schema with the scanner framework and provides context information for the reader. Scans use either a "dynamic" or a defined schema.

Regardless of the schema type, the result of building the schema is a result set loader used to prepare batches for use in the query. The reader can simply read all columns, allowing the framework to discard unwanted values. Or for efficiency, the reader can check the column metadata to determine if a column is projected, and if not, then don't even read the column from the input source.

Defined Schema

If defined, the execution plan provides the output schema (presumably computed from an accurate metadata source.) The reader must populate the proscribed rows, performing column type conversions as needed. The reader can determine if the schema is defined by calling hasOutputSchema().

At present, the scan framework filters the "provided schema" against the project list so that this class presents only the actual output schema. Future versions may do the filtering in the planner, but the result for readers will be the same either way.

Dynamic Schema

A dynamic schema occurs when the plan does not specify a schema. Drill is unique in its support for "schema on read" in the sense that Drill does not know the schema until the reader defines it at scan time.

The reader and scan framework coordinate to form the output schema. The reader offers the columns it has available. The scan framework uses the projection list to decide which to accept. Either way the scan framework provides a column reader for the column (returning a do-nothing "dummy" reader if the column is unprojected.)

With a dynamic schema, readers offer a schema in one of two ways:

The reader provides the table schema in one of two ways: early schema or late schema. Either way, the project list from the physical plan determines which table columns are materialized and which are not. Readers are provided for all table columns for readers that must read sequentially, but only the materialized columns are written to value vectors.

Early Dynamic Schema

Some readers can determine the source schema at the start of a scan. For example, a CSV file has headers, a Parquet file has footers, both of which define a schema. This case is called "early schema." The reader fefines the schema by calling tableSchema(TupleMetadata) to provide the known schema.

Late Dynamic Schema

Other readers don't know the input schema until the reader actually reads the data. For example, JSON typically has no schema, but does have sufficient structure (name/value pairs) to infer one.

The late schema reader calls RowSetLoader#addColumn() to add each column as it is discovered during the scan.

Note that, to avoid schema conflicts, a late schema reader must define the full set of columns in the first batch, and must stick to that schema for all subsequent batches. This allows the reader to look one batch ahead to learn the columns.

Drill, however, cannot predict the future. Without a defined schema, downstream operators cannot know which columns might appear later in the scan, with which types. Today this is a strong guideline. Future versions may enforce this rule.

Method Summary

Modifier and Type	Method	Description
void	batchSize(int maxRecordsPerBatch)	Set the preferred batch size (which may be overridden by the result set loader in order to limit vector or batch size.)
ResultSetLoader	build()	Build the schema, plan the required projections and static columns and return a loader used to populate value vectors.
OperatorContext	context()
CustomErrorContext	errorContext()	Returns the error context to use for this reader: either the parent or the reader-specific context set in setErrorContext(CustomErrorContext).
TupleMetadata	inputSchema()	Returns the reader input schema: the schema which describes the set of columns this reader should produce.
boolean	isProjectionEmpty()	Report whether the projection list is empty, as occurs in two cases: SELECT COUNT(*) ... -- empty project. SELECT a, b FROM table(c d) -- disjoint project.
CustomErrorContext	parentErrorContext()	The context to use as a parent when creating a custom context.
ProjectedColumn	projectionFor(String colName)
TupleMetadata	providedSchema()	Returns the provided schema, if defined.
void	schemaIsComplete(boolean isComplete)
void	setErrorContext(CustomErrorContext context)	Specify an advanced error context which allows the reader to fill in custom context values.
void	tableSchema(TupleMetadata schema)
void	tableSchema(TupleMetadata schema, boolean isComplete)	Specify the table schema if this is an early-schema reader.
String	userName()	Name of the user running the query.

Method Details

context

OperatorContext context()

parentErrorContext

CustomErrorContext parentErrorContext()

The context to use as a parent when creating a custom context.

(Obtain the error context for this reader from the ResultSetLoader.

setErrorContext

void setErrorContext(CustomErrorContext context)

Specify an advanced error context which allows the reader to fill in custom context values.

errorContext

CustomErrorContext errorContext()

Returns the error context to use for this reader: either the parent or the reader-specific context set in setErrorContext(CustomErrorContext).

userName

String userName()

Name of the user running the query.

isProjectionEmpty

boolean isProjectionEmpty()

Report whether the projection list is empty, as occurs in two cases:

• SELECT COUNT(*) ... -- empty project.

SELECT a, b FROM table(c d) -- disjoint project.