org.apache.drill.exec.store

Class ColumnExplorer

java.lang.Object

org.apache.drill.exec.store.ColumnExplorer

public class ColumnExplorer
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ColumnExplorer.ImplicitFileColumn
static class	ColumnExplorer.ImplicitFileColumns Columns that give information from where file data comes from.
static class	ColumnExplorer.ImplicitInternalFileColumns Columns that give internal information about file or its parts.

Constructor Summary

Constructors

Constructor and Description
ColumnExplorer(OptionManager optionManager) Constructor for using the column explorer to probe existing columns in the ProjectRecordBatch.
ColumnExplorer(OptionManager optionManager, List<SchemaPath> columns) Helper class that encapsulates logic for sorting out columns between actual table columns, partition columns and implicit file columns.

Method Summary

Modifier and Type	Method and Description
boolean	containsImplicitColumns() Checks if current column selection contains implicit columns.
boolean	containsPartitionColumns() Checks if current column selection contains partition columns.
static List<String>	getImplicitColumnsNames(SchemaConfig schemaConfig) Returns list with implicit column names taken from specified SchemaConfig.
static String	getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column, org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.FileSystem fs) Returns implicit column value for specified implicit file column.
static String	getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column, org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.FileSystem fs, Integer index, Long start, Long length) Returns implicit column value for specified implicit file column.
static List<ColumnExplorer.ImplicitFileColumn>	getImplicitFileColumns() Returns list of implicit file columns which includes all elements from ColumnExplorer.ImplicitFileColumns, ColumnExplorer.ImplicitInternalFileColumns.LAST_MODIFIED_TIME and ColumnExplorer.ImplicitInternalFileColumns.USE_METADATA columns.
static List<String>	getPartitionColumnNames(FileSelection selection, ColumnNamesOptions columnNamesOptions) Returns list with partition column names.
static int	getPartitionDepth(FileSelection selection)
List<SchemaPath>	getTableColumns()
static Map<String,ColumnExplorer.ImplicitFileColumns>	initImplicitFileColumns(OptionManager optionManager) Creates case insensitive map with implicit file columns as keys and appropriate ImplicitFileColumns enum as values
static Map<String,ColumnExplorer.ImplicitInternalFileColumns>	initImplicitInternalFileColumns(OptionManager optionManager) Creates case insensitive map with implicit internal file columns as keys and appropriate ImplicitFileColumns enum as values
boolean	isImplicitOrInternalFileColumn(String name) Checks whether given column is implicit or internal.
static boolean	isPartitionColumn(OptionManager optionManager, SchemaPath column) Checks if given column is partition or not.
static boolean	isPartitionColumn(String partitionDesignator, String path) Checks if given column is partition or not.
boolean	isStarQuery()
static List<String>	listPartitionValues(org.apache.hadoop.fs.Path file, org.apache.hadoop.fs.Path root, boolean hasDirsOnly) Compares root and file path to determine directories that are present in the file path but absent in root.
static String[]	parsePartitions(org.apache.hadoop.fs.Path file, org.apache.hadoop.fs.Path root, boolean hasDirsOnly) Low-level parse of partitions, returned as a string array.
Map<String,String>	populateColumns(org.apache.hadoop.fs.Path filePath, List<String> partitionValues, boolean includeFileImplicitColumns, org.apache.hadoop.fs.FileSystem fs) Creates map with implicit and internal columns where key is column name, value is columns actual value.
Map<String,String>	populateColumns(org.apache.hadoop.fs.Path filePath, List<String> partitionValues, boolean includeFileImplicitColumns, org.apache.hadoop.fs.FileSystem fs, int index, long start, long length) Creates map with implicit and internal columns where key is column name, value is columns actual value.
Map<String,String>	populateImplicitColumns(org.apache.hadoop.fs.Path filePath, List<String> partitionValues, boolean includeFileImplicitColumns) Creates map with implicit columns where key is column name, value is columns actual value.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ColumnExplorer

public ColumnExplorer(OptionManager optionManager,
                      List<SchemaPath> columns)

Helper class that encapsulates logic for sorting out columns between actual table columns, partition columns and implicit file columns. Also populates map with implicit columns names as keys and their values

ColumnExplorer

public ColumnExplorer(OptionManager optionManager)

Constructor for using the column explorer to probe existing columns in the ProjectRecordBatch.

Method Detail

initImplicitFileColumns

public static Map<String,ColumnExplorer.ImplicitFileColumns> initImplicitFileColumns(OptionManager optionManager)

Creates case insensitive map with implicit file columns as keys and appropriate ImplicitFileColumns enum as values

initImplicitInternalFileColumns

public static Map<String,ColumnExplorer.ImplicitInternalFileColumns> initImplicitInternalFileColumns(OptionManager optionManager)

Creates case insensitive map with implicit internal file columns as keys and appropriate ImplicitFileColumns enum as values

getImplicitColumnsNames

public static List<String> getImplicitColumnsNames(SchemaConfig schemaConfig)

Returns list with implicit column names taken from specified SchemaConfig.

Parameters:

schemaConfig - the source of session options values.

Returns:

list with implicit column names.

isPartitionColumn

public static boolean isPartitionColumn(OptionManager optionManager,
                                        SchemaPath column)

Checks if given column is partition or not.

Parameters:

optionManager - options

column - column

Returns:

true if given column is partition, false otherwise

isPartitionColumn

public static boolean isPartitionColumn(String partitionDesignator,
                                        String path)

Checks if given column is partition or not.

Parameters:

partitionDesignator - partition designator

path - column path

Returns:

true if given column is partition, false otherwise

isImplicitOrInternalFileColumn

public boolean isImplicitOrInternalFileColumn(String name)

Checks whether given column is implicit or internal.

Parameters:

name - name of the column to check

Returns:

true if given column is implicit or internal, false otherwise

getPartitionColumnNames

public static List<String> getPartitionColumnNames(FileSelection selection,
                                                   ColumnNamesOptions columnNamesOptions)

Returns list with partition column names. For the case when table has several levels of nesting, max level is chosen.

Parameters:

selection - the source of file paths

columnNamesOptions - the source of session option value for partition column label

Returns:

list with partition column names.

getPartitionDepth

public static int getPartitionDepth(FileSelection selection)

populateImplicitColumns

public Map<String,String> populateImplicitColumns(org.apache.hadoop.fs.Path filePath,
                                                  List<String> partitionValues,
                                                  boolean includeFileImplicitColumns)

Creates map with implicit columns where key is column name, value is columns actual value. This map contains partition and implicit file columns (if requested). Partition columns names are formed based in partition designator and value index.

Parameters:

filePath - file path, used to populate file implicit columns

partitionValues - list of partition values

includeFileImplicitColumns - if file implicit columns should be included into the result

Returns:

implicit columns map

populateColumns

public Map<String,String> populateColumns(org.apache.hadoop.fs.Path filePath,
                                          List<String> partitionValues,
                                          boolean includeFileImplicitColumns,
                                          org.apache.hadoop.fs.FileSystem fs)

Creates map with implicit and internal columns where key is column name, value is columns actual value. This map contains partition, implicit and internal file columns (if requested). Partition columns names are formed based in partition designator and value index.

Parameters:

filePath - file path, used to populate file implicit columns

partitionValues - list of partition values

includeFileImplicitColumns - if file implicit columns should be included into the result

fs - file system

Returns:

implicit columns map

populateColumns

public Map<String,String> populateColumns(org.apache.hadoop.fs.Path filePath,
                                          List<String> partitionValues,
                                          boolean includeFileImplicitColumns,
                                          org.apache.hadoop.fs.FileSystem fs,
                                          int index,
                                          long start,
                                          long length)

Creates map with implicit and internal columns where key is column name, value is columns actual value. This map contains partition, implicit and internal file columns (if requested). Partition columns names are formed based in partition designator and value index.

Parameters:

filePath - file path, used to populate file implicit columns

partitionValues - list of partition values

includeFileImplicitColumns - if file implicit columns should be included into the result

fs - file system

index - index of row group to populate

start - start of row group to populate

length - length of row group to populate

Returns:

implicit columns map

getImplicitColumnValue

public static String getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column,
                                            org.apache.hadoop.fs.Path filePath,
                                            org.apache.hadoop.fs.FileSystem fs,
                                            Integer index,
                                            Long start,
                                            Long length)

Returns implicit column value for specified implicit file column.

Parameters:

column - implicit file column

filePath - file path, used to populate file implicit columns

fs - file system

index - row group index

start - row group start

length - row group length

Returns:

implicit column value for specified implicit file column

getImplicitColumnValue

public static String getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column,
                                            org.apache.hadoop.fs.Path filePath,
                                            org.apache.hadoop.fs.FileSystem fs)

Returns implicit column value for specified implicit file column.

Parameters:

column - implicit file column

filePath - file path

fs - file system

Returns:

implicit column value for specified implicit file column

getImplicitFileColumns

public static List<ColumnExplorer.ImplicitFileColumn> getImplicitFileColumns()

Returns list of implicit file columns which includes all elements from ColumnExplorer.ImplicitFileColumns, ColumnExplorer.ImplicitInternalFileColumns.LAST_MODIFIED_TIME and ColumnExplorer.ImplicitInternalFileColumns.USE_METADATA columns.

Returns:

list of implicit file columns

listPartitionValues

public static List<String> listPartitionValues(org.apache.hadoop.fs.Path file,
                                               org.apache.hadoop.fs.Path root,
                                               boolean hasDirsOnly)

Compares root and file path to determine directories that are present in the file path but absent in root. Example: root - a/b/c, file - a/b/c/d/e/0_0_0.parquet, result - d/e. Stores different directory names in the list in successive order.

Parameters:

file - file path

root - root directory

hasDirsOnly - whether it is file or directory

Returns:

list of directory names

parsePartitions

public static String[] parsePartitions(org.apache.hadoop.fs.Path file,
                                       org.apache.hadoop.fs.Path root,
                                       boolean hasDirsOnly)

Low-level parse of partitions, returned as a string array. Returns a null array for invalid values.

Parameters:

file - file path

root - root directory

hasDirsOnly - whether it is file or directory

Returns:

array of directory names, or null if the arguments are invalid

isStarQuery

public boolean isStarQuery()

getTableColumns

public List<SchemaPath> getTableColumns()

containsPartitionColumns

public boolean containsPartitionColumns()

Checks if current column selection contains partition columns.

Returns:

true if partition columns are present, false otherwise

containsImplicitColumns

public boolean containsImplicitColumns()

Checks if current column selection contains implicit columns.

Returns:

true if implicit columns are present, false otherwise