Class AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B extends AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B>>
java.lang.Object
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B>
- Direct Known Subclasses:
AbstractParquetGroupScan.RowGroupScanFilterer
- Enclosing class:
- AbstractGroupScanWithMetadata<P extends TableMetadataProvider>
protected abstract static class AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B extends AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B>>
extends Object
This class is responsible for filtering different metadata levels.
-
Field Summary
Modifier and TypeFieldDescriptionprotected FunctionLookupContext
protected Map<org.apache.hadoop.fs.Path,
FileMetadata> protected LogicalExpression
protected int
protected boolean
protected NonInterestingColumnsMetadata
protected MetadataType
protected List<PartitionMetadata>
protected Map<org.apache.hadoop.fs.Path,
SegmentMetadata> protected final AbstractGroupScanWithMetadata<? extends TableMetadataProvider>
protected TableMetadata
protected TupleMetadata
protected UdfUtilities
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionabstract AbstractGroupScanWithMetadata<?>
build()
Constructs required implementation ofAbstractGroupScanWithMetadata
with filtered metadata.context
(FunctionLookupContext context) files
(Map<org.apache.hadoop.fs.Path, FileMetadata> files) filterAndGetMetadata
(Set<SchemaPath> schemaPathsInExpr, Iterable<T> metadataList, FilterPredicate<?> filterPredicate, OptionManager optionManager) Produces filtering of specified metadata using specified filter expression and returns filtered metadata.filterExpression
(LogicalExpression filterExpression) protected void
filterFileMetadata
(OptionManager optionManager, FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at file level.protected void
filterPartitionMetadata
(OptionManager optionManager, FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at partition level.protected void
filterSegmentMetadata
(OptionManager optionManager, FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at segment level.protected void
filterTableMetadata
(FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at table level.Map<org.apache.hadoop.fs.Path,
FileMetadata> getFiles()
protected B
getFiltered
(OptionManager optionManager, FilterPredicate<?> filterPredicate) Produces filtering of metadata and returnsAbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer
to construct resulting group scan.protected <T extends Metadata>
Map<SchemaPath,ColumnStatistics<?>> getImplicitColumnStatistics
(OptionManager optionManager, T metadata, Map<SchemaPath, ColumnStatistics<?>> columnsStatistics) Map<org.apache.hadoop.fs.Path,
SegmentMetadata> boolean
limit
(int maxRecords) matching
(boolean matchAllMetadata) nonInterestingColumns
(NonInterestingColumnsMetadata nonInterestingColumns) overflow
(MetadataType overflowLevel) partitions
(List<PartitionMetadata> partitions) protected List<PartitionMetadata>
pruneForSegments
(List<PartitionMetadata> metadataToPrune, Map<org.apache.hadoop.fs.Path, SegmentMetadata> filteredSegmentMetadata) Removes metadata which does not belong to any of partitions in metadata list.protected static <T extends BaseMetadata & LocationProvider>
Map<org.apache.hadoop.fs.Path,T> pruneForSegments
(Map<org.apache.hadoop.fs.Path, T> metadataToPrune, Map<org.apache.hadoop.fs.Path, SegmentMetadata> filteredSegmentMetadata) Removes metadata which does not belong to any of partitions in metadata list.schema
(TupleMetadata tableSchema) segments
(Map<org.apache.hadoop.fs.Path, SegmentMetadata> segments) protected abstract B
self()
table
(TableMetadata tableMetadata) udfUtilities
(UdfUtilities udfUtilities)
-
Field Details
-
source
-
matchAllMetadata
protected boolean matchAllMetadata -
tableMetadata
-
partitions
-
segments
-
files
-
nonInterestingColumnsMetadata
-
filterExpression
-
tableSchema
-
udfUtilities
-
context
-
limit
protected int limit -
overflowLevel
-
-
Constructor Details
-
GroupScanWithMetadataFilterer
-
-
Method Details
-
build
Constructs required implementation ofAbstractGroupScanWithMetadata
with filtered metadata.- Returns:
- implementation of
AbstractGroupScanWithMetadata
with filtered metadata
-
table
-
partitions
-
segments
-
nonInterestingColumns
-
files
-
limit
-
matching
-
overflow
-
filterExpression
-
schema
-
udfUtilities
-
context
-
isMatchAllMetadata
public boolean isMatchAllMetadata() -
getTableMetadata
-
getPartitions
-
getSegments
-
getFiles
-
getOverflowLevel
-
getFiltered
Produces filtering of metadata and returnsAbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer
to construct resulting group scan.- Parameters:
optionManager
- option managerfilterPredicate
- filter expression- Returns:
- this instance with filtered metadata
-
filterTableMetadata
protected void filterTableMetadata(FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at table level.- Parameters:
filterPredicate
- filter expressionschemaPathsInExpr
- columns used in filter expression
-
filterSegmentMetadata
protected void filterSegmentMetadata(OptionManager optionManager, FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at segment level.- Parameters:
optionManager
- option managerfilterPredicate
- filter expressionschemaPathsInExpr
- columns used in filter expression
-
filterPartitionMetadata
protected void filterPartitionMetadata(OptionManager optionManager, FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at partition level.- Parameters:
optionManager
- option managerfilterPredicate
- filter expressionschemaPathsInExpr
- columns used in filter expression
-
filterFileMetadata
protected void filterFileMetadata(OptionManager optionManager, FilterPredicate<?> filterPredicate, Set<SchemaPath> schemaPathsInExpr) Produces filtering of metadata at file level.- Parameters:
optionManager
- option managerfilterPredicate
- filter expressionschemaPathsInExpr
- columns used in filter expression
-
pruneForSegments
protected static <T extends BaseMetadata & LocationProvider> Map<org.apache.hadoop.fs.Path,T> pruneForSegments(Map<org.apache.hadoop.fs.Path, T> metadataToPrune, Map<org.apache.hadoop.fs.Path, SegmentMetadata> filteredSegmentMetadata) Removes metadata which does not belong to any of partitions in metadata list.- Type Parameters:
T
- type of metadata to filter- Parameters:
metadataToPrune
- list of metadata which should be prunedfilteredSegmentMetadata
- list of segment metadata which was pruned- Returns:
- map with metadata which belongs to pruned partitions
-
pruneForSegments
protected List<PartitionMetadata> pruneForSegments(List<PartitionMetadata> metadataToPrune, Map<org.apache.hadoop.fs.Path, SegmentMetadata> filteredSegmentMetadata) Removes metadata which does not belong to any of partitions in metadata list.- Parameters:
metadataToPrune
- list of partition metadata which should be prunedfilteredSegmentMetadata
- list of segment metadata which was pruned- Returns:
- list with metadata which belongs to pruned partitions
-
filterAndGetMetadata
public <T extends Metadata> List<T> filterAndGetMetadata(Set<SchemaPath> schemaPathsInExpr, Iterable<T> metadataList, FilterPredicate<?> filterPredicate, OptionManager optionManager) Produces filtering of specified metadata using specified filter expression and returns filtered metadata.- Type Parameters:
T
- type of metadata to filter- Parameters:
schemaPathsInExpr
- columns used in filter expressionmetadataList
- metadata to filterfilterPredicate
- filter expressionoptionManager
- option manager- Returns:
- filtered metadata
-
getImplicitColumnStatistics
protected <T extends Metadata> Map<SchemaPath,ColumnStatistics<?>> getImplicitColumnStatistics(OptionManager optionManager, T metadata, Map<SchemaPath, ColumnStatistics<?>> columnsStatistics) -
self
-