org.apache.drill.exec.physical.config

Class AbstractMuxExchange

java.lang.Object

org.apache.drill.exec.physical.base.AbstractBase

org.apache.drill.exec.physical.base.AbstractSingle

org.apache.drill.exec.physical.base.AbstractExchange

org.apache.drill.exec.physical.config.AbstractMuxExchange

All Implemented Interfaces:

Iterable<PhysicalOperator>, GraphValue<PhysicalOperator>, Exchange, PhysicalOperator

Direct Known Subclasses:

OrderedMuxExchange, UnorderedMuxExchange

public abstract class AbstractMuxExchange
extends AbstractExchange

Multiplexing Exchange (MuxExchange) is used when results from multiple minor fragments belonging to the same major fragment running on a node need to be collected at one fragment on the same node before distributing the results further. This helps when the sender that is distributing the results has overhead that is proportional to the number of sender instances. An example of such sender is PartitionSender. Each instance of PartitionSender allocates "r" buffers where "r" is the number of receivers. Ex. Drillbit A is assigned 10 minor fragments belonging to the same major fragment. Each of these fragments has a PartitionSender instance which is sending data to 300 receivers. Each PartitionSender needs 300 buffers, so total of 10*300 buffers are needed. With MuxExchange, all 10 fragments send the data directly (without partitioning) to MuxExchange which uses the PartitionSender to partition the incoming data and distribute to receivers. MuxExchange has only one instance per Drillbit per major fragment which means only one instance of PartitionSender per Drillbit per major fragment. With MuxExchange total number of buffers used by PartitionSender for the 10 fragments is 300 instead of earlier number 10*300.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.drill.exec.physical.base.Exchange

Exchange.ParallelizationDependency

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap<Integer,MinorFragmentEndpoint>	receiverToSenderMapping
protected Map<Integer,MinorFragmentEndpoint>	senderToReceiverMapping

Fields inherited from class org.apache.drill.exec.physical.base.AbstractExchange

receiverLocations, receiverMajorFragmentId, senderLocations, senderMajorFragmentId

Fields inherited from class org.apache.drill.exec.physical.base.AbstractSingle

child

Fields inherited from class org.apache.drill.exec.physical.base.AbstractBase

INIT_ALLOCATION, initialAllocation, MAX_ALLOCATION, maxAllocation, userName

Constructor Summary

Constructors

Constructor and Description
AbstractMuxExchange(PhysicalOperator child)

Method Summary

Modifier and Type	Method and Description
protected void	createSenderReceiverMapping()
ParallelizationInfo	getReceiverParallelizationInfo(List<CoordinationProtos.DrillbitEndpoint> senderFragmentEndpoints) Default receiver parallelization width range is [1, Integer.MAX_VALUE] and affinity to nodes where sender fragments are running.
Sender	getSender(int minorFragmentId, PhysicalOperator child) Get the Sender associated with the given minorFragmentId.
protected List<MinorFragmentEndpoint>	getSenders(int minorFragmentId)

Methods inherited from class org.apache.drill.exec.physical.base.AbstractExchange

accept, getDefaultAffinityMap, getOperatorType, getParallelizationDependency, getReceiverMemory, getSenderMemory, getSenderParallelizationInfo, isExecutable, setupReceivers, setupReceivers, setupSenders, setupSenders

Methods inherited from class org.apache.drill.exec.physical.base.AbstractSingle

getChild, getNewWithChild, getNewWithChildren, iterator

Methods inherited from class org.apache.drill.exec.physical.base.AbstractBase

accept, getCost, getInitialAllocation, getMaxAllocation, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.physical.base.Exchange

getChild, getReceiver

Methods inherited from interface org.apache.drill.exec.physical.base.PhysicalOperator

getCost, getInitialAllocation, getMaxAllocation, getNewWithChildren, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from interface org.apache.drill.common.graph.GraphValue

accept

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Field Detail

senderToReceiverMapping

protected Map<Integer,MinorFragmentEndpoint> senderToReceiverMapping

receiverToSenderMapping

protected org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap<Integer,MinorFragmentEndpoint> receiverToSenderMapping

Constructor Detail

AbstractMuxExchange

public AbstractMuxExchange(PhysicalOperator child)

Method Detail

getReceiverParallelizationInfo

public ParallelizationInfo getReceiverParallelizationInfo(List<CoordinationProtos.DrillbitEndpoint> senderFragmentEndpoints)

Description copied from class: AbstractExchange

Default receiver parallelization width range is [1, Integer.MAX_VALUE] and affinity to nodes where sender fragments are running.

Specified by:

getReceiverParallelizationInfo in interface Exchange

Overrides:

getReceiverParallelizationInfo in class AbstractExchange

Parameters:

senderFragmentEndpoints - Endpoints assigned to receiver fragment if available, otherwise an empty list.

Returns:

Receiver ParallelizationInfo.

getSender

public Sender getSender(int minorFragmentId,
                        PhysicalOperator child)

Description copied from interface: Exchange

Get the Sender associated with the given minorFragmentId. Cannot be called until after setupSenders() and setupReceivers() have been called.

Parameters:

minorFragmentId - The minor fragment id, must be in the range [0, fragment.width).

child - The feeding node for the requested sender.

Returns:

The materialized sender for the given arguments.

getSenders

protected final List<MinorFragmentEndpoint> getSenders(int minorFragmentId)

createSenderReceiverMapping

protected void createSenderReceiverMapping()