org.apache.drill.exec.physical.impl.join

Class HashJoinBatch

java.lang.Object

org.apache.drill.exec.record.AbstractRecordBatch<T>

org.apache.drill.exec.record.AbstractBinaryRecordBatch<HashJoinPOP>

org.apache.drill.exec.physical.impl.join.HashJoinBatch

All Implemented Interfaces:

AutoCloseable, Iterable<VectorWrapper<?>>, RowKeyJoin, CloseableRecordBatch, RecordBatch, VectorAccessible

public class HashJoinBatch
extends AbstractBinaryRecordBatch<HashJoinPOP>
implements RowKeyJoin

Implements the runtime execution for the Hash-Join operator supporting INNER, LEFT OUTER, RIGHT OUTER, and FULL OUTER joins

This implementation splits the incoming Build side rows into multiple Partitions, thus allowing spilling of some of these partitions to disk if memory gets tight. Each partition is implemented as a HashPartition. After the build phase is over, in the most general case, some of the partitions were spilled, and the others are in memory. Each of the partitions in memory would get a HashTable built.

Next the Probe side is read, and each row is key matched with a Build partition. If that partition is in memory, then the key is used to probe and perform the join, and the results are added to the outgoing batch. But if that build side partition was spilled, then the matching Probe size partition is spilled as well.

After all the Probe side was processed, we are left with pairs of spilled partitions. Then each pair is processed individually (that Build partition should be smaller than the original, hence likely fit whole into memory to allow probing; if not -- see below).

Processing of each spilled pair is EXACTLY like processing the original Build/Probe incomings. (As a fact, the innerNext() method calls itself recursively !!). Thus the spilled build partition is read and divided into new partitions, which in turn may spill again (and again...). The code tracks these spilling "cycles". Normally any such "again" (i.e. cycle of 2 or greater) is a waste, indicating that the number of partitions chosen was too small.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HashJoinBatch.HashJoinSpilledPartition This holds information about the spilled partitions for the build and probe side.
class	HashJoinBatch.HashJoinUpdater
static class	HashJoinBatch.Metric

Nested classes/interfaces inherited from class org.apache.drill.exec.record.AbstractRecordBatch

AbstractRecordBatch.BatchState

Nested classes/interfaces inherited from interface org.apache.drill.exec.physical.impl.join.RowKeyJoin

RowKeyJoin.RowKeyJoinState

Nested classes/interfaces inherited from interface org.apache.drill.exec.record.RecordBatch

RecordBatch.IterOutcome

Field Summary

Fields inherited from class org.apache.drill.exec.record.AbstractBinaryRecordBatch

batchMemoryManager, left, LEFT_INDEX, leftUpstream, numInputs, right, RIGHT_INDEX, rightUpstream

Fields inherited from class org.apache.drill.exec.record.AbstractRecordBatch

batchStatsContext, container, context, oContext, state, stats, unionTypeEnabled

Fields inherited from interface org.apache.drill.exec.record.RecordBatch

MAX_BATCH_ROW_COUNT

Constructor Summary

Constructors

Constructor and Description
HashJoinBatch(HashJoinPOP popConfig, FragmentContext context, RecordBatch left, RecordBatch right) The constructor

Method Summary

Modifier and Type	Method and Description
protected void	buildSchema()
protected void	cancelIncoming()
void	close()
void	dump() Perform dump of this batch's state to logs.
RecordBatch.IterOutcome	executeBuildPhase() Execute the BUILD phase; first read incoming and split rows into partitions; may decide to spill some of the partitions
AbstractRecordBatch.BatchState	getBatchState() Get the current BatchState (this is useful when performing row key join)
HashJoinMemoryCalculator	getCalculatorImpl() Determines the memory calculator to use.
int	getRecordCount() Get the number of records.
RowKeyJoin.RowKeyJoinState	getRowKeyJoinState() Get the current RowKeyJoinState.
boolean	hasRowKeyBatch() Is the next batch of row keys ready to be returned
RecordBatch.IterOutcome	innerNext()
boolean	isSpilledInner(int part)
String	makeDebugString() This creates a string that summarizes the memory usage of the operator.
org.apache.commons.lang3.tuple.Pair<ValueVector,Integer>	nextRowKeyBatch() Get the hash table iterator that is created for the build side of the hash join if this hash join was instantiated as a row-key join.
void	setBatchState(AbstractRecordBatch.BatchState newState) Set the BatchState (this is useful when performing row key join)
void	setRowKeyJoinState(RowKeyJoin.RowKeyJoinState newState) Set the RowKeyJoinState (this is useful for maintaining state for row key join algorithm)
HashJoinProbe	setupHashJoinProbe()
void	updateMetrics()

Methods inherited from class org.apache.drill.exec.record.AbstractBinaryRecordBatch

checkForEarlyFinish, getBatchMemoryManager, prefetchFirstBatchFromBothSides, updateBatchMemoryManagerStats, verifyOutcomeToSetBatchState

Methods inherited from class org.apache.drill.exec.record.AbstractRecordBatch

cancel, checkContinue, getContainer, getContext, getOutgoingContainer, getPopConfig, getRecordBatchStatsContext, getSchema, getSelectionVector2, getSelectionVector4, getValueAccessorById, getValueVectorId, getWritableBatch, isRecordBatchStatsLoggingEnabled, iterator, next, next, next, schemaChangeException, schemaChangeException

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

HashJoinBatch

public HashJoinBatch(HashJoinPOP popConfig,
                     FragmentContext context,
                     RecordBatch left,
                     RecordBatch right)
              throws OutOfMemoryException

The constructor

Parameters:

popConfig -

context -

left - -- probe/outer side incoming input

right - -- build/iner side incoming input

Throws:

OutOfMemoryException

Method Detail

getRecordCount

public int getRecordCount()

Description copied from interface: VectorAccessible

Get the number of records.

Specified by:

getRecordCount in interface VectorAccessible

Returns:

number of records

buildSchema

protected void buildSchema()

Overrides:

buildSchema in class AbstractRecordBatch<HashJoinPOP>

getCalculatorImpl

public HashJoinMemoryCalculator getCalculatorImpl()

Determines the memory calculator to use. If maxNumBatches is configured simple batch counting is used to spill. Otherwise memory calculations are used to determine when to spill.

Returns:

The memory calculator to use.

innerNext

public RecordBatch.IterOutcome innerNext()

Specified by:

innerNext in class AbstractRecordBatch<HashJoinPOP>

executeBuildPhase

public RecordBatch.IterOutcome executeBuildPhase()
                                          throws SchemaChangeException

Execute the BUILD phase; first read incoming and split rows into partitions; may decide to spill some of the partitions

Returns:

Returns an RecordBatch.IterOutcome if a termination condition is reached. Otherwise returns null.

Throws:

SchemaChangeException

isSpilledInner

public boolean isSpilledInner(int part)

makeDebugString

public String makeDebugString()

This creates a string that summarizes the memory usage of the operator.

Returns:

A memory dump string.

nextRowKeyBatch

public org.apache.commons.lang3.tuple.Pair<ValueVector,Integer> nextRowKeyBatch()

Get the hash table iterator that is created for the build side of the hash join if this hash join was instantiated as a row-key join.

Specified by:

nextRowKeyBatch in interface RowKeyJoin

Returns:

hash table iterator or null if this hash join was not a row-key join or if it was a row-key join but the build has not yet completed.

hasRowKeyBatch

public boolean hasRowKeyBatch()

Description copied from interface: RowKeyJoin

Is the next batch of row keys ready to be returned

Specified by:

hasRowKeyBatch in interface RowKeyJoin

Returns:

True if ready, false if not

getBatchState

public AbstractRecordBatch.BatchState getBatchState()

Description copied from interface: RowKeyJoin

Get the current BatchState (this is useful when performing row key join)

Specified by:

getBatchState in interface RowKeyJoin

setBatchState

public void setBatchState(AbstractRecordBatch.BatchState newState)

Description copied from interface: RowKeyJoin

Set the BatchState (this is useful when performing row key join)

Specified by:

setBatchState in interface RowKeyJoin

cancelIncoming

protected void cancelIncoming()

Overrides:

cancelIncoming in class AbstractBinaryRecordBatch<HashJoinPOP>

updateMetrics

public void updateMetrics()

setRowKeyJoinState

public void setRowKeyJoinState(RowKeyJoin.RowKeyJoinState newState)

Description copied from interface: RowKeyJoin

Set the RowKeyJoinState (this is useful for maintaining state for row key join algorithm)

Specified by:

setRowKeyJoinState in interface RowKeyJoin

getRowKeyJoinState

public RowKeyJoin.RowKeyJoinState getRowKeyJoinState()

Description copied from interface: RowKeyJoin

Get the current RowKeyJoinState.

Specified by:

getRowKeyJoinState in interface RowKeyJoin

close

public void close()

Specified by:

close in interface AutoCloseable

Overrides:

close in class AbstractRecordBatch<HashJoinPOP>

setupHashJoinProbe

public HashJoinProbe setupHashJoinProbe()

dump

public void dump()

Description copied from interface: RecordBatch

Perform dump of this batch's state to logs.

Specified by:

dump in interface RecordBatch