org.apache.drill.exec.store.dfs

Class FileSelection

java.lang.Object

org.apache.drill.exec.store.dfs.FileSelection

All Implemented Interfaces:

DrillTableSelection

Direct Known Subclasses:

IcebergMetadataFileSelection

public class FileSelection
extends Object
implements DrillTableSelection

Jackson serializable description of a file selection.

Field Summary

Fields

Modifier and Type	Field and Description
org.apache.hadoop.fs.Path	cacheFileRoot root path for the metadata cache file (if any)
List<org.apache.hadoop.fs.Path>	files
org.apache.hadoop.fs.Path	selectionRoot root path for the selections

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	FileSelection(FileSelection selection) Copy constructor for convenience.
	FileSelection(List<org.apache.hadoop.fs.FileStatus> statuses, List<org.apache.hadoop.fs.Path> files, org.apache.hadoop.fs.Path selectionRoot) Creates a selection out of given file statuses/files and selection root.
	FileSelection(List<org.apache.hadoop.fs.FileStatus> statuses, List<org.apache.hadoop.fs.Path> files, org.apache.hadoop.fs.Path selectionRoot, org.apache.hadoop.fs.Path cacheFileRoot, boolean wasAllPartitionsPruned)
	FileSelection(List<org.apache.hadoop.fs.FileStatus> statuses, List<org.apache.hadoop.fs.Path> files, org.apache.hadoop.fs.Path selectionRoot, org.apache.hadoop.fs.Path cacheFileRoot, boolean wasAllPartitionsPruned, org.apache.drill.exec.store.dfs.FileSelection.StatusType dirStatus)

Method Summary

Modifier and Type	Method and Description
static void	checkBackPaths(String parent, String combinedPath, String subpath) Check if the path is a valid sub path under the parent after removing backpaths.
boolean	containsDirectories(DrillFileSystem fs)
static FileSelection	create(DrillFileSystem fs, String parent, String path, boolean allowAccessOutsideWorkspace)
static FileSelection	create(List<org.apache.hadoop.fs.FileStatus> statuses, List<org.apache.hadoop.fs.Path> files, org.apache.hadoop.fs.Path root)
static FileSelection	create(List<org.apache.hadoop.fs.FileStatus> statuses, List<org.apache.hadoop.fs.Path> files, org.apache.hadoop.fs.Path root, org.apache.hadoop.fs.Path cacheFileRoot, boolean wasAllPartitionsPruned) Creates a selection with the given file statuses/files and selection root.
static FileSelection	createFromDirectories(List<org.apache.hadoop.fs.Path> dirPaths, FileSelection selection, org.apache.hadoop.fs.Path cacheFileRoot)
String	digest() The digest of the selection represented by the implementation.
org.apache.hadoop.fs.Path	getCacheFileRoot()
org.apache.drill.exec.store.dfs.FileSelection.StatusType	getDirStatus()
List<org.apache.hadoop.fs.Path>	getFiles()
List<org.apache.hadoop.fs.FileStatus>	getFileStatuses()
org.apache.hadoop.fs.FileStatus	getFirstPath(DrillFileSystem fs)
MetadataContext	getMetaContext()
org.apache.hadoop.fs.Path	getSelectionRoot()
List<org.apache.hadoop.fs.FileStatus>	getStatuses(DrillFileSystem fs)
boolean	hadWildcard()
boolean	isEmptyDirectory()
boolean	isExpandedFully()
boolean	isExpandedPartial()
FileSelection	minusDirectories(DrillFileSystem fs)
void	setEmptyDirectoryStatus() Setting emptyDirectory as true allows to identify this selectionRoot as an empty directory
void	setExpandedFully()
void	setExpandedPartial()
void	setHadWildcard(boolean wc)
void	setMetaContext(MetadataContext context)
boolean	supportsDirPruning()
String	toString()
boolean	wasAllPartitionsPruned()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

files

public List<org.apache.hadoop.fs.Path> files

selectionRoot

public final org.apache.hadoop.fs.Path selectionRoot

root path for the selections

cacheFileRoot

public final org.apache.hadoop.fs.Path cacheFileRoot

root path for the metadata cache file (if any)

Constructor Detail

FileSelection

public FileSelection(List<org.apache.hadoop.fs.FileStatus> statuses,
                     List<org.apache.hadoop.fs.Path> files,
                     org.apache.hadoop.fs.Path selectionRoot)

Creates a selection out of given file statuses/files and selection root.

Parameters:

statuses - list of file statuses

files - list of files

selectionRoot - root path for selections

FileSelection

public FileSelection(List<org.apache.hadoop.fs.FileStatus> statuses,
                     List<org.apache.hadoop.fs.Path> files,
                     org.apache.hadoop.fs.Path selectionRoot,
                     org.apache.hadoop.fs.Path cacheFileRoot,
                     boolean wasAllPartitionsPruned)

FileSelection

public FileSelection(List<org.apache.hadoop.fs.FileStatus> statuses,
                     List<org.apache.hadoop.fs.Path> files,
                     org.apache.hadoop.fs.Path selectionRoot,
                     org.apache.hadoop.fs.Path cacheFileRoot,
                     boolean wasAllPartitionsPruned,
                     org.apache.drill.exec.store.dfs.FileSelection.StatusType dirStatus)

FileSelection

protected FileSelection(FileSelection selection)

Copy constructor for convenience.

Method Detail

getSelectionRoot

public org.apache.hadoop.fs.Path getSelectionRoot()

getStatuses

public List<org.apache.hadoop.fs.FileStatus> getStatuses(DrillFileSystem fs)
                                                  throws IOException

Throws:

IOException

getFiles

public List<org.apache.hadoop.fs.Path> getFiles()

containsDirectories

public boolean containsDirectories(DrillFileSystem fs)
                            throws IOException

Throws:

IOException

minusDirectories

public FileSelection minusDirectories(DrillFileSystem fs)
                               throws IOException

Throws:

IOException

getFirstPath

public org.apache.hadoop.fs.FileStatus getFirstPath(DrillFileSystem fs)
                                             throws IOException

Throws:

IOException

setExpandedFully

public void setExpandedFully()

isExpandedFully

public boolean isExpandedFully()

setExpandedPartial

public void setExpandedPartial()

isExpandedPartial

public boolean isExpandedPartial()

getDirStatus

public org.apache.drill.exec.store.dfs.FileSelection.StatusType getDirStatus()

wasAllPartitionsPruned

public boolean wasAllPartitionsPruned()

create

public static FileSelection create(DrillFileSystem fs,
                                   String parent,
                                   String path,
                                   boolean allowAccessOutsideWorkspace)
                            throws IOException

Throws:

IOException

create

public static FileSelection create(List<org.apache.hadoop.fs.FileStatus> statuses,
                                   List<org.apache.hadoop.fs.Path> files,
                                   org.apache.hadoop.fs.Path root,
                                   org.apache.hadoop.fs.Path cacheFileRoot,
                                   boolean wasAllPartitionsPruned)

Creates a selection with the given file statuses/files and selection root.

Parameters:

statuses - list of file statuses

files - list of files

root - root path for selections

cacheFileRoot - root path for metadata cache (null for no metadata cache)

Returns:

null if creation of FileSelection fails with an IllegalArgumentException otherwise a new selection.

See Also:

FileSelection(List, List, Path)

create

public static FileSelection create(List<org.apache.hadoop.fs.FileStatus> statuses,
                                   List<org.apache.hadoop.fs.Path> files,
                                   org.apache.hadoop.fs.Path root)

createFromDirectories

public static FileSelection createFromDirectories(List<org.apache.hadoop.fs.Path> dirPaths,
                                                  FileSelection selection,
                                                  org.apache.hadoop.fs.Path cacheFileRoot)

checkBackPaths

public static void checkBackPaths(String parent,
                                  String combinedPath,
                                  String subpath)

Check if the path is a valid sub path under the parent after removing backpaths. Throw an exception if it is not. We pass subpath in as a parameter only for the error message

Parameters:

parent - The parent path (the workspace directory).

combinedPath - The workspace directory and (relative) subpath path combined.

subpath - For error message only, the subpath

getFileStatuses

public List<org.apache.hadoop.fs.FileStatus> getFileStatuses()

supportsDirPruning

public boolean supportsDirPruning()

setHadWildcard

public void setHadWildcard(boolean wc)

hadWildcard

public boolean hadWildcard()

getCacheFileRoot

public org.apache.hadoop.fs.Path getCacheFileRoot()

setMetaContext

public void setMetaContext(MetadataContext context)

getMetaContext

public MetadataContext getMetaContext()

isEmptyDirectory

public boolean isEmptyDirectory()

Returns:

true if this selectionRoot points to an empty directory, false otherwise

setEmptyDirectoryStatus

public void setEmptyDirectoryStatus()

Setting emptyDirectory as true allows to identify this selectionRoot as an empty directory

digest

public String digest()

Description copied from interface: DrillTableSelection

The digest of the selection represented by the implementation. The selections that accompany Tables can modify the contained dataset, e.g. a file selection can restrict to a subset of the available data and a format selection can include options that affect the behaviour of the underlying reader. Two scans will end up being considered identical during logical planning if their digests are the same so selection implementations should override this method so that exactly those scans that really are identical (in terms of the data they produce) have matching digests.

Specified by:

digest in interface DrillTableSelection

Returns:

this selection's digest, normally a string built from its properties.

toString

public String toString()

Overrides:

toString in class Object