Package org.apache.drill.exec.physical.impl.join

Class HashJoinMemoryCalculatorImpl.PostBuildCalculationsImpl

java.lang.Object

org.apache.drill.exec.physical.impl.join.HashJoinMemoryCalculatorImpl.PostBuildCalculationsImpl

All Implemented Interfaces:

HashJoinMemoryCalculator.PostBuildCalculations, HashJoinStateCalculator<HashJoinMemoryCalculator>

Enclosing class:

HashJoinMemoryCalculatorImpl

public static class HashJoinMemoryCalculatorImpl.PostBuildCalculationsImpl
extends Object
implements HashJoinMemoryCalculator.PostBuildCalculations

In this state, we need to make sure there is enough room to spill probe side batches, if spilling is necessary. If there is not enough room, we have to evict build side partitions. If we don't have to evict build side partitions in this state, then we are done. If we do have to evict build side partitions then we have to recursively repeat the process.

Lifecycle

• Step 1: Call initialize(boolean). This gives the HashJoinStateCalculator additional information it needs to compute memory requirements.
• Step 2: Call shouldSpill(). This tells you which build side partitions need to be spilled in order to make room for probing.
• Step 3: Call next(). After you are done probing and partitioning the probe side, get the next calculator.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	MIN_RECORDS_PER_PARTITION_BATCH_PROBE

Constructor Summary

Constructors

Constructor

Description

PostBuildCalculationsImpl(boolean firstCycle, BatchSizePredictor probeSizePredictor, long memoryAvailable, long maxOutputBatchSize, int maxBatchNumRecordsProbe, int recordsPerPartitionBatchProbe, HashJoinMemoryCalculator.PartitionStatSet buildPartitionStatSet, Map<String,Long> keySizes, HashTableSizeCalculator hashTableSizeCalculator, HashJoinHelperSizeCalculator hashJoinHelperSizeCalculator, double fragmentationFactor, double safetyFactor, double loadFactor, boolean reserveHash, boolean semiJoin)

Method Summary

Modifier and Type	Method	Description
static long	calculateReservedMemory(int numSpilledPartitions, long maxProbeBatchSize, long maxOutputBatchSize, long partitionProbeBatchSize)
static int	computeProbeRecordsPerBatch(long memoryAvailable, int numPartitions, int defaultProbeRecordsPerBatch, int minProbeRecordsPerBatch, long maxProbeBatchSize, long maxOutputBatchSize, long defaultPartitionProbeBatchSize)
long	getConsumedMemory()
long	getIncomingProbeBatchReservedSpace()
long	getPartitionProbeBatchSize()
int	getProbeRecordsPerBatch()
HashJoinState	getState()	The current HashJoinState corresponding to this calculator.
void	initialize(boolean probeEmpty)	Initializes the calculator with additional information needed.
String	makeDebugString()
HashJoinMemoryCalculator	next()	Signifies that the current state is complete and returns the next HashJoinStateCalculator.
boolean	shouldSpill()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MIN_RECORDS_PER_PARTITION_BATCH_PROBE

public static final int MIN_RECORDS_PER_PARTITION_BATCH_PROBE

See Also:

• Constant Field Values

Constructor Details

PostBuildCalculationsImpl

public PostBuildCalculationsImpl(boolean firstCycle,
 BatchSizePredictor probeSizePredictor,
 long memoryAvailable,
 long maxOutputBatchSize,
 int maxBatchNumRecordsProbe,
 int recordsPerPartitionBatchProbe,
 HashJoinMemoryCalculator.PartitionStatSet buildPartitionStatSet,
 Map<String,Long> keySizes,
 HashTableSizeCalculator hashTableSizeCalculator,
 HashJoinHelperSizeCalculator hashJoinHelperSizeCalculator,
 double fragmentationFactor,
 double safetyFactor,
 double loadFactor,
 boolean reserveHash,
 boolean semiJoin)

Method Details

initialize

public void initialize(boolean probeEmpty)

Description copied from interface: HashJoinMemoryCalculator.PostBuildCalculations

Initializes the calculator with additional information needed.

Specified by:

initialize in interface HashJoinMemoryCalculator.PostBuildCalculations

Parameters:

probeEmpty - True if the probe is empty. False otherwise.

getProbeRecordsPerBatch

public int getProbeRecordsPerBatch()

Specified by:

getProbeRecordsPerBatch in interface HashJoinMemoryCalculator.PostBuildCalculations

getIncomingProbeBatchReservedSpace

public long getIncomingProbeBatchReservedSpace()

getPartitionProbeBatchSize

public long getPartitionProbeBatchSize()

getConsumedMemory

public long getConsumedMemory()

computeProbeRecordsPerBatch

public static int computeProbeRecordsPerBatch(long memoryAvailable,
 int numPartitions,
 int defaultProbeRecordsPerBatch,
 int minProbeRecordsPerBatch,
 long maxProbeBatchSize,
 long maxOutputBatchSize,
 long defaultPartitionProbeBatchSize)

calculateReservedMemory

public static long calculateReservedMemory(int numSpilledPartitions,
 long maxProbeBatchSize,
 long maxOutputBatchSize,
 long partitionProbeBatchSize)

shouldSpill

public boolean shouldSpill()

Specified by:

shouldSpill in interface HashJoinMemoryCalculator.PostBuildCalculations

next

public HashJoinMemoryCalculator next()

Description copied from interface: HashJoinStateCalculator

Signifies that the current state is complete and returns the next HashJoinStateCalculator. Returns null in the case where there is no next state.

Specified by:

next in interface HashJoinStateCalculator<HashJoinMemoryCalculator>

Returns:

The next HashJoinStateCalculator or null if this was the last state.

getState

public HashJoinState getState()

Description copied from interface: HashJoinStateCalculator

The current HashJoinState corresponding to this calculator.

Specified by:

getState in interface HashJoinStateCalculator<HashJoinMemoryCalculator>

Returns:

makeDebugString

public String makeDebugString()

Specified by:

makeDebugString in interface HashJoinMemoryCalculator.PostBuildCalculations