Package org.apache.drill.exec.store.easy.text

Class TextFormatPlugin

java.lang.Object

org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin<TextFormatConfig>

org.apache.drill.exec.store.easy.text.TextFormatPlugin

All Implemented Interfaces:

FormatPlugin

public class TextFormatPlugin
extends EasyFormatPlugin<TextFormatConfig>

Text format plugin for CSV and other delimited text formats. Allows use of a "provided schema", including using table properties on that schema to override "static" ("or default") properties defined in the plugin config. Allows, say, having ".csv" files in which some have no headers (the default) and some do have headers (as specified via table properties in the provided schema.)

Makes use of the scan framework and the result set loader mechanism to allow tight control of the size of produced batches (as well as to support provided schema.)

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

EasyFormatPlugin.EasyFormatConfig, EasyFormatPlugin.EasyFormatConfigBuilder, EasyFormatPlugin.ScanFrameworkVersion

Field Summary

Fields

Modifier and Type	Field	Description
static final String	COMMENT_CHAR_PROP
static final String	DELIMITER_PROP
static final String	HAS_HEADERS_PROP
static final String	LINE_DELIM_PROP
static final int	MAX_CHARS_PER_COLUMN
static final int	MAXIMUM_NUMBER_COLUMNS
static final char	NULL_CHAR
static final String	PARSE_UNESCAPED_QUOTES_PROP
static final String	QUOTE_ESCAPE_PROP
static final String	QUOTE_PROP
static final String	SKIP_FIRST_LINE_PROP
static final String	TEXT_PREFIX
static final String	TRIM_WHITESPACE_PROP
static final String	WRITER_OPERATOR_TYPE

Fields inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

formatConfig

Constructor Summary

Constructors

Constructor	Description
TextFormatPlugin(String name, DrillbitContext context, org.apache.hadoop.conf.Configuration fsConf, StoragePluginConfig config, TextFormatConfig formatPluginConfig)

Method Summary

Modifier and Type	Method	Description
protected void	configureScan(FileScanLifecycleBuilder builder, EasySubScan scan)	Configure an EVF (v2) scan, which must at least include the factory to create readers.
AbstractGroupScan	getGroupScan(String userName, FileSelection selection, List<SchemaPath> columns, MetadataProviderManager metadataProviderManager)
AbstractGroupScan	getGroupScan(String userName, FileSelection selection, List<SchemaPath> columns, OptionManager options, MetadataProviderManager metadataProviderManager)
RecordWriter	getRecordWriter(FragmentContext context, EasyWriter writer)
protected ScanStats	getScanStats(PlannerSettings settings, EasyGroupScan scan)

Methods inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

easyConfig, frameworkBuilder, getConfig, getContext, getFsConf, getGroupScan, getMatcher, getName, getOptimizerRules, getReaderBatch, getReaderOperatorType, getRecordReader, getStatisticsRecordWriter, getStorageConfig, getWriter, getWriterBatch, getWriterOperatorType, initScanBuilder, isBlockSplittable, isCompressible, isStatisticsRecordWriter, newBatchReader, readStatistics, scanVersion, supportsAutoPartitioning, supportsFileImplicitColumns, supportsLimitPushdown, supportsPushDown, supportsRead, supportsStatistics, supportsWrite, writeStatistics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.store.dfs.FormatPlugin

getGroupScan, getOptimizerRules

Field Details

MAXIMUM_NUMBER_COLUMNS

public static final int MAXIMUM_NUMBER_COLUMNS

See Also:

• Constant Field Values

MAX_CHARS_PER_COLUMN

public static final int MAX_CHARS_PER_COLUMN

See Also:

• Constant Field Values

NULL_CHAR

public static final char NULL_CHAR

See Also:

• Constant Field Values

TEXT_PREFIX

public static final String TEXT_PREFIX

HAS_HEADERS_PROP

public static final String HAS_HEADERS_PROP

SKIP_FIRST_LINE_PROP

public static final String SKIP_FIRST_LINE_PROP

DELIMITER_PROP

public static final String DELIMITER_PROP

COMMENT_CHAR_PROP

public static final String COMMENT_CHAR_PROP

QUOTE_PROP

public static final String QUOTE_PROP

QUOTE_ESCAPE_PROP

public static final String QUOTE_ESCAPE_PROP

LINE_DELIM_PROP

public static final String LINE_DELIM_PROP

TRIM_WHITESPACE_PROP

public static final String TRIM_WHITESPACE_PROP

PARSE_UNESCAPED_QUOTES_PROP

public static final String PARSE_UNESCAPED_QUOTES_PROP

WRITER_OPERATOR_TYPE

public static final String WRITER_OPERATOR_TYPE

See Also:

• Constant Field Values

Constructor Details

TextFormatPlugin

public TextFormatPlugin(String name,
 DrillbitContext context,
 org.apache.hadoop.conf.Configuration fsConf,
 StoragePluginConfig config,
 TextFormatConfig formatPluginConfig)

Method Details

getGroupScan

public AbstractGroupScan getGroupScan(String userName,
 FileSelection selection,
 List<SchemaPath> columns,
 MetadataProviderManager metadataProviderManager)
                               throws IOException

Specified by:

getGroupScan in interface FormatPlugin

Overrides:

getGroupScan in class EasyFormatPlugin<TextFormatConfig>

Throws:

IOException

getGroupScan

public AbstractGroupScan getGroupScan(String userName,
 FileSelection selection,
 List<SchemaPath> columns,
 OptionManager options,
 MetadataProviderManager metadataProviderManager)
                               throws IOException

Throws:

IOException

configureScan

protected void configureScan(FileScanLifecycleBuilder builder,
 EasySubScan scan)

Description copied from class: EasyFormatPlugin

Configure an EVF (v2) scan, which must at least include the factory to create readers.

Overrides:

configureScan in class EasyFormatPlugin<TextFormatConfig>

Parameters:

builder - the builder with default options already set, and which allows the plugin implementation to set others

getRecordWriter

public RecordWriter getRecordWriter(FragmentContext context,
 EasyWriter writer)
                             throws IOException

Overrides:

getRecordWriter in class EasyFormatPlugin<TextFormatConfig>

Throws:

IOException

getScanStats

protected ScanStats getScanStats(PlannerSettings settings,
 EasyGroupScan scan)

Overrides:

getScanStats in class EasyFormatPlugin<TextFormatConfig>