org.apache.drill.exec.physical.impl.scan.file

Class ImplicitColumnManager

java.lang.Object

org.apache.drill.exec.physical.impl.scan.file.ImplicitColumnManager

All Implemented Interfaces:

MetadataManager, ReaderLevelProjection.ReaderProjectionResolver, VectorSource

public class ImplicitColumnManager
extends Object
implements MetadataManager, ReaderLevelProjection.ReaderProjectionResolver, VectorSource

Manages the insertion of file metadata (AKA "implicit" and partition) columns. Parses the file metadata columns from the projection list. Creates and loads the vectors that hold the data. If running in legacy mode, inserts partition columns when the query contains a wildcard. Supports renaming the columns via session options.

The lifecycle is that the manager is given the set of files for this scan operator so it can determine the partition depth. (Note that different scans may not agree on the depth. This is a known issue with Drill's implementation.)

Then, at the start of the scan, all projection columns are parsed. This class picks out the file metadata columns.

On each file (on each reader), the columns are "resolved." Here, that means that the columns are filled in with actual values based on the present file.

This is the successor to ColumnExplorer.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ImplicitColumnManager.ImplicitColumnOptions

Field Summary

Fields

Modifier and Type	Field and Description
static int	AUTO_PARTITION_DEPTH Automatically compute partition depth from files.
protected Map<String,FileMetadataColumnDefn>	fileMetadataColIndex
protected List<FileMetadataColumnDefn>	implicitColDefns
protected String	partitionDesignator

Constructor Summary

Constructors

Constructor and Description
ImplicitColumnManager(OptionSet optionManager, ImplicitColumnManager.ImplicitColumnOptions config) Constructor for ImplicitColumnManager for managing the insertion of file metadata (AKA "implicit" and partition) columns.
ImplicitColumnManager(OptionSet optionManager, ImplicitColumnManager.ImplicitColumnOptions config, org.apache.hadoop.fs.FileSystem fs) Constructor for ImplicitColumnManager for managing the insertion of file metadata (AKA "implicit" and partition) columns.

Method Summary

Modifier and Type	Method and Description
void	bind(ResultVectorCache vectorCache)
void	close() Event indicating the end of a scan.
void	define() Define (materialize) the columns which this manager represents.
void	endFile() Event indicating the end of a file (or other data source.)
FileMetadata	fileMetadata(org.apache.hadoop.fs.Path filePath)
List<FileMetadataColumnDefn>	fileMetadataColDefns()
boolean	hasImplicitCols()
void	load(int rowCount) Load data into the custom columns, if needed (such as for null or implicit columns.)
List<MetadataColumn>	metadataColumns()
protected ImplicitColumnManager.ImplicitColumnOptions	options()
int	partitionCount()
String	partitionName(int partition)
ScanLevelProjection.ScanProjectionParser	projectionParser() Returns the file metadata column parser that: Picks out the file metadata and partition columns, Inserts partition columns for a wildcard query, if the option to do so is set.
boolean	resolveColumn(ColumnProjection col, ResolvedTuple tuple, TupleMetadata tableSchema) Resolves metadata columns to concrete, materialized columns with the proper value for the present file.
ReaderLevelProjection.ReaderProjectionResolver	resolver()
void	startFile(org.apache.hadoop.fs.Path filePath)
void	startResolution()
ValueVector	vector(int index)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AUTO_PARTITION_DEPTH

public static final int AUTO_PARTITION_DEPTH

Automatically compute partition depth from files. Use only for testing!

See Also:

Constant Field Values

partitionDesignator

protected final String partitionDesignator

implicitColDefns

protected final List<FileMetadataColumnDefn> implicitColDefns

fileMetadataColIndex

protected final Map<String,FileMetadataColumnDefn> fileMetadataColIndex

Constructor Detail

ImplicitColumnManager

public ImplicitColumnManager(OptionSet optionManager,
                             ImplicitColumnManager.ImplicitColumnOptions config,
                             org.apache.hadoop.fs.FileSystem fs)

Constructor for ImplicitColumnManager for managing the insertion of file metadata (AKA "implicit" and partition) columns.

Parameters:

optionManager - access to the options for this query; used too look up custom names for the metadata columns

config - implicit column options

fs - file system

ImplicitColumnManager

public ImplicitColumnManager(OptionSet optionManager,
                             ImplicitColumnManager.ImplicitColumnOptions config)

Constructor for ImplicitColumnManager for managing the insertion of file metadata (AKA "implicit" and partition) columns.

Parameters:

optionManager - access to the options for this query; used too look up custom names for the metadata columns

config - implicit column options

Method Detail

options

protected ImplicitColumnManager.ImplicitColumnOptions options()

bind

public void bind(ResultVectorCache vectorCache)

Specified by:

bind in interface MetadataManager

projectionParser

public ScanLevelProjection.ScanProjectionParser projectionParser()

Returns the file metadata column parser that:

• Picks out the file metadata and partition columns,
• Inserts partition columns for a wildcard query, if the option to do so is set.

Specified by:

projectionParser in interface MetadataManager

See Also:

ImplicitColumnManager.ImplicitColumnOptions.useLegacyWildcardExpansion

fileMetadata

public FileMetadata fileMetadata(org.apache.hadoop.fs.Path filePath)

hasImplicitCols

public boolean hasImplicitCols()

partitionName

public String partitionName(int partition)

fileMetadataColDefns

public List<FileMetadataColumnDefn> fileMetadataColDefns()

startFile

public void startFile(org.apache.hadoop.fs.Path filePath)

resolver

public ReaderLevelProjection.ReaderProjectionResolver resolver()

Specified by:

resolver in interface MetadataManager

define

public void define()

Description copied from interface: MetadataManager

Define (materialize) the columns which this manager represents.

Specified by:

define in interface MetadataManager

load

public void load(int rowCount)

Description copied from interface: MetadataManager

Load data into the custom columns, if needed (such as for null or implicit columns.)

Specified by:

load in interface MetadataManager

Parameters:

rowCount - number of rows read into a batch.

close

public void close()

Description copied from interface: MetadataManager

Event indicating the end of a scan.

Specified by:

close in interface MetadataManager

startResolution

public void startResolution()

Specified by:

startResolution in interface ReaderLevelProjection.ReaderProjectionResolver

endFile

public void endFile()

Description copied from interface: MetadataManager

Event indicating the end of a file (or other data source.)

Specified by:

endFile in interface MetadataManager

resolveColumn

public boolean resolveColumn(ColumnProjection col,
                             ResolvedTuple tuple,
                             TupleMetadata tableSchema)

Resolves metadata columns to concrete, materialized columns with the proper value for the present file.

Specified by:

resolveColumn in interface ReaderLevelProjection.ReaderProjectionResolver

vector

public ValueVector vector(int index)

Specified by:

vector in interface VectorSource

partitionCount

public int partitionCount()

metadataColumns

public List<MetadataColumn> metadataColumns()