public class SmoothingProjection extends ReaderLevelProjection
Consider this an experimental mechanism. The hope was that, with clever techniques, we could "smooth over" some of the issues that cause schema change events in Drill. As it turned out, however, creating this mechanism revealed that it is not possible, even in theory, to handle most schema changes because of the time dimension:
What this mechanism can do is make decisions based on history: when a column appears, we can adjust its type a bit to try to avoid an unnecessary change. For example, if a prior file in this scan saw `foo` as nullable Varchar, but the present file has the column as requied Varchar, we can use the more general nullable form. But, again, the "can't predict the future" bites us: we can handle a nullable-to-required column change, but not visa-versa.
What this mechanism will tell the careful reader is that the only general solution to the schema-change problem is to now the full schema up front: for the planner to be told the schema and to communicate that schema to all readers so that all readers agree on the final schema.
When that is done, the techniques shown here can be used to adjust any per-file variation of schema to match the up-front schema.
|Modifier and Type||Field and Description|
|Constructor and Description|
|Modifier and Type||Method and Description|
protected final List<MaterializedField> rewrittenFields
public SmoothingProjection(ScanLevelProjection scanProj, TupleMetadata tableSchema, ResolvedTuple priorSchema, ResolvedTuple outputTuple, List<ReaderLevelProjection.ReaderProjectionResolver> resolvers) throws SchemaSmoother.IncompatibleSchemaException
public List<MaterializedField> revisedTableSchema()
Copyright © 1970 The Apache Software Foundation. All rights reserved.