Package org.apache.drill.exec.store.parquet

Class BaseParquetMetadataProvider

java.lang.Object

org.apache.drill.exec.store.parquet.BaseParquetMetadataProvider

All Implemented Interfaces:

ParquetMetadataProvider, TableMetadataProvider

Direct Known Subclasses:

DeltaParquetTableMetadataProvider, HiveParquetTableMetadataProvider, ParquetTableMetadataProviderImpl

public abstract class BaseParquetMetadataProvider
extends Object
implements ParquetMetadataProvider

Implementation of ParquetMetadataProvider which contains base methods for obtaining metadata from parquet statistics.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	BaseParquetMetadataProvider.Builder<T extends BaseParquetMetadataProvider.Builder<T>>

Field Summary

Fields

Modifier and Type	Field	Description
protected final List<ReadEntryWithPath>	entries
protected Set<org.apache.hadoop.fs.Path>	fileSet
static final Object	NULL_VALUE	HashBasedTable cannot contain nulls, used this object to represent null values.
protected MetadataBase.ParquetTableMetadataBase	parquetTableMetadata
protected final ParquetReaderConfig	readerConfig
protected TupleMetadata	schema
protected DrillStatsTable	statsTable
protected final org.apache.hadoop.fs.Path	tableLocation
protected final String	tableName

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BaseParquetMetadataProvider(BaseParquetMetadataProvider.Builder<?> builder)

Method Summary

Modifier and Type	Method	Description
boolean	checkMetadataVersion()	Whether metadata actuality should be checked.
List<ReadEntryWithPath>	getEntries()	Returns list of ReadEntryWithPath instances which represents paths to files to be scanned.
FileMetadata	getFileMetadata(org.apache.hadoop.fs.Path location)	Returns FileMetadata instance which corresponds to metadata of file for specified location.
Set<org.apache.hadoop.fs.Path>	getFileSet()	Returns list of file locations for table.
List<FileMetadata>	getFilesForPartition(PartitionMetadata partition)	Returns list of FileMetadata instances which belongs to specified partitions.
Map<org.apache.hadoop.fs.Path,FileMetadata>	getFilesMetadataMap()	Returns map of FileMetadata instances which provides metadata for specific file and its columns.
List<org.apache.hadoop.fs.Path>	getLocations()	Returns list of file paths which belong to current table.
NonInterestingColumnsMetadata	getNonInterestingColumnsMetadata()	Returns NonInterestingColumnsMetadata instance which provides metadata for non-interesting columns.
List<SchemaPath>	getPartitionColumns()	Returns list of partition columns for table from this TableMetadataProvider.
List<PartitionMetadata>	getPartitionMetadata(SchemaPath columnName)	Returns list of PartitionMetadata instances which corresponds to partitions for specified column and provides metadata for specific partitions and its columns.
List<PartitionMetadata>	getPartitionsMetadata()	Returns list of PartitionMetadata instances which provides metadata for specific partitions and its columns.
List<RowGroupMetadata>	getRowGroupsMeta()	Returns list of RowGroupMetadata instances which provides metadata for specific row group and its columns.
org.apache.drill.shaded.guava.com.google.common.collect.Multimap<org.apache.hadoop.fs.Path,RowGroupMetadata>	getRowGroupsMetadataMap()	Returns multimap of RowGroupMetadata instances which provides metadata for specific row group and its columns mapped to their locations.
Map<org.apache.hadoop.fs.Path,SegmentMetadata>	getSegmentsMetadataMap()	Returns map of SegmentMetadata instances which provides metadata for segment and its columns.
TableMetadata	getTableMetadata()	Returns TableMetadata instance which provides metadata for table and columns metadata.
protected void	init(BaseParquetMetadataProvider metadataProvider)
void	initializeMetadata()	Method which initializes all metadata kinds to get rid of parquetTableMetadata.
protected abstract void	initInternal()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

NULL_VALUE

public static final Object NULL_VALUE

HashBasedTable cannot contain nulls, used this object to represent null values.

entries

protected final List<ReadEntryWithPath> entries

readerConfig

protected final ParquetReaderConfig readerConfig

tableName

protected final String tableName

tableLocation

protected final org.apache.hadoop.fs.Path tableLocation

parquetTableMetadata

protected MetadataBase.ParquetTableMetadataBase parquetTableMetadata

fileSet

protected Set<org.apache.hadoop.fs.Path> fileSet

schema

protected TupleMetadata schema

statsTable

protected DrillStatsTable statsTable

Constructor Details

BaseParquetMetadataProvider

protected BaseParquetMetadataProvider(BaseParquetMetadataProvider.Builder<?> builder)

Method Details

init

protected void init(BaseParquetMetadataProvider metadataProvider)
             throws IOException

Throws:

IOException

initializeMetadata

public void initializeMetadata()

Method which initializes all metadata kinds to get rid of parquetTableMetadata. Once deserialization and serialization from/into metastore classes is done, this method should be removed to allow lazy initialization.

getNonInterestingColumnsMetadata

public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata()

Description copied from interface: TableMetadataProvider

Returns NonInterestingColumnsMetadata instance which provides metadata for non-interesting columns.

Specified by:

getNonInterestingColumnsMetadata in interface TableMetadataProvider

Returns:

NonInterestingColumnsMetadata instance

getTableMetadata

public TableMetadata getTableMetadata()

Description copied from interface: TableMetadataProvider

Returns TableMetadata instance which provides metadata for table and columns metadata.

Specified by:

getTableMetadata in interface TableMetadataProvider

Returns:

TableMetadata instance

getPartitionColumns

public List<SchemaPath> getPartitionColumns()

Description copied from interface: TableMetadataProvider

Returns list of partition columns for table from this TableMetadataProvider.

Specified by:

getPartitionColumns in interface TableMetadataProvider

Returns:

list of partition columns

getPartitionsMetadata

public List<PartitionMetadata> getPartitionsMetadata()

Description copied from interface: TableMetadataProvider

Returns list of PartitionMetadata instances which provides metadata for specific partitions and its columns.

Specified by:

getPartitionsMetadata in interface TableMetadataProvider

Returns:

list of PartitionMetadata instances

getPartitionMetadata

public List<PartitionMetadata> getPartitionMetadata(SchemaPath columnName)

Description copied from interface: TableMetadataProvider

Returns list of PartitionMetadata instances which corresponds to partitions for specified column and provides metadata for specific partitions and its columns.

Specified by:

getPartitionMetadata in interface TableMetadataProvider

Returns:

list of PartitionMetadata instances which corresponds to partitions for specified column

getFileMetadata

public FileMetadata getFileMetadata(org.apache.hadoop.fs.Path location)

Description copied from interface: TableMetadataProvider

Returns FileMetadata instance which corresponds to metadata of file for specified location.

Specified by:

getFileMetadata in interface TableMetadataProvider

Parameters:

location - location of the file

Returns:

FileMetadata instance which corresponds to metadata of file for specified location

getFilesForPartition

public List<FileMetadata> getFilesForPartition(PartitionMetadata partition)

Description copied from interface: TableMetadataProvider

Returns list of FileMetadata instances which belongs to specified partitions.

Specified by:

getFilesForPartition in interface TableMetadataProvider

Parameters:

partition - partition which

Returns:

list of FileMetadata instances which belongs to specified partitions

getSegmentsMetadataMap

public Map<org.apache.hadoop.fs.Path,SegmentMetadata> getSegmentsMetadataMap()

Description copied from interface: TableMetadataProvider

Returns map of SegmentMetadata instances which provides metadata for segment and its columns.

Specified by:

getSegmentsMetadataMap in interface TableMetadataProvider

Returns:

map of SegmentMetadata instances

getFilesMetadataMap

public Map<org.apache.hadoop.fs.Path,FileMetadata> getFilesMetadataMap()

Description copied from interface: TableMetadataProvider

Returns map of FileMetadata instances which provides metadata for specific file and its columns.

Specified by:

getFilesMetadataMap in interface TableMetadataProvider

Returns:

map of FileMetadata instances

getEntries

public List<ReadEntryWithPath> getEntries()

Description copied from interface: ParquetMetadataProvider

Returns list of ReadEntryWithPath instances which represents paths to files to be scanned.

Specified by:

getEntries in interface ParquetMetadataProvider

Returns:

list of ReadEntryWithPath instances whith file paths

getFileSet

public Set<org.apache.hadoop.fs.Path> getFileSet()

Description copied from interface: ParquetMetadataProvider

Returns list of file locations for table.

Specified by:

getFileSet in interface ParquetMetadataProvider

Returns:

list of file locations for table

getRowGroupsMeta

public List<RowGroupMetadata> getRowGroupsMeta()

Description copied from interface: ParquetMetadataProvider

Returns list of RowGroupMetadata instances which provides metadata for specific row group and its columns.

Specified by:

getRowGroupsMeta in interface ParquetMetadataProvider

Returns:

list of RowGroupMetadata instances

getLocations

public List<org.apache.hadoop.fs.Path> getLocations()

Description copied from interface: ParquetMetadataProvider

Returns list of file paths which belong to current table.

Specified by:

getLocations in interface ParquetMetadataProvider

Returns:

list of file paths

getRowGroupsMetadataMap

public org.apache.drill.shaded.guava.com.google.common.collect.Multimap<org.apache.hadoop.fs.Path,RowGroupMetadata> getRowGroupsMetadataMap()

Description copied from interface: ParquetMetadataProvider

Returns multimap of RowGroupMetadata instances which provides metadata for specific row group and its columns mapped to their locations.

Specified by:

getRowGroupsMetadataMap in interface ParquetMetadataProvider

Returns:

multimap of RowGroupMetadata instances

checkMetadataVersion

public boolean checkMetadataVersion()

Description copied from interface: TableMetadataProvider

Whether metadata actuality should be checked.

Specified by:

checkMetadataVersion in interface TableMetadataProvider

Returns:

true if metadata actuality should be checked

initInternal

protected abstract void initInternal()
                              throws IOException

Throws:

IOException