public class MemoryAllocationUtilities extends Object
Constructor and Description |
---|
MemoryAllocationUtilities() |
Modifier and Type | Method and Description |
---|---|
static long |
computeOperatorMemory(OptionSet optionManager,
long maxAllocPerNode,
int opCount)
Compute per-operator memory based on the computed per-node memory, the
number of operators, and the computed number of fragments (which house
the operators.) Enforces a floor on the amount of memory per operator.
|
static long |
computeQueryMemory(DrillConfig config,
OptionSet optionManager,
long directMemory)
Per-node memory calculations based on a number of constraints.
|
static List<PhysicalOperator> |
getBufferedOperators(List<PhysicalOperator> operators,
QueryContext queryContext) |
static void |
setupBufferedMemoryAllocations(PhysicalPlan plan,
QueryContext queryContext) |
static void |
setupBufferedOpsMemoryAllocations(boolean planHasMemory,
List<PhysicalOperator> bufferedOperators,
QueryContext queryContext)
Helper method to setup Memory Allocations
|
public static void setupBufferedMemoryAllocations(PhysicalPlan plan, QueryContext queryContext)
public static List<PhysicalOperator> getBufferedOperators(List<PhysicalOperator> operators, QueryContext queryContext)
public static void setupBufferedOpsMemoryAllocations(boolean planHasMemory, List<PhysicalOperator> bufferedOperators, QueryContext queryContext)
Plan the memory for buffered operators (the only ones that can spill in this release) based on assumptions. These assumptions are the amount of memory per node to give to each query and the number of sort operators per node.
The reason the total memory is an assumption is that we have know knowledge of the number of queries that can run, so we need the user to tell use that information by configuring the amount of memory to be assumed available to each query.
The number of sorts per node could be calculated, but we instead simply take the worst case: the maximum per-query, per-node parallization and assume that all sorts appear in all fragments — a gross oversimplification, but one that Drill has long made.
since this method can be used in multiple places adding it in this class rather than keeping it in Foreman
planHasMemory
- defines the memory planning needs to be done or not.
generally skipped when the plan contains memory allocation.bufferedOperators
- list of buffered operators in the plan.queryContext
- context of the query.public static long computeOperatorMemory(OptionSet optionManager, long maxAllocPerNode, int opCount)
optionManager
- system option managermaxAllocPerNode
- computed query memory per nodeopCount
- number of buffering operators in this querypublic static long computeQueryMemory(DrillConfig config, OptionSet optionManager, long directMemory)
Factored out into a separate method to allow unit testing.
config
- Drill configoptionManager
- system optionsdirectMemory
- amount of direct memoryCopyright © 1970 The Apache Software Foundation. All rights reserved.