Skip navigation links

Package org.apache.drill.exec.resourcemgr.config

This package contains the configuration components of ResourceManagement feature in Drill.

See: Description

Package org.apache.drill.exec.resourcemgr.config Description

This package contains the configuration components of ResourceManagement feature in Drill. ResourceManagement will have it's own configuration file supporting the similar hierarchy of files as supported by Drill's current configuration and supports HOCON format. All the supported files for ResourceManagement is listed in ConfigConstants. However whether the feature is enabled/disabled is still controlled by a configuration ExecConstants.RM_ENABLED available in the Drill's main configuration file. The rm config files will be parsed and loaded only when the feature is enabled. The configuration is a hierarchical tree ResourcePoolTree of ResourcePool. At the top will be the root pool which represents the entire resources (only memory in version 1) which is available to ResourceManager to use for admitting queries. It is assumed that all the nodes in the Drill cluster is homogeneous and given same amount of memory resources. The root pool can be further divided into child ResourcePools to divide the resources among multiple child pools. Each child pool get's a resource share from it's parent resource pool. In theory there is no limit on the number of ResourcePools that can be configured to divide the cluster resources.

In addition to other parameters defined later root ResourcePool also supports a configuration ResourcePoolTreeImpl.ROOT_POOL_QUEUE_SELECTION_POLICY_KEY which helps to select exactly one leaf pool out of all the possible options available for a query. For details please see of QueueSelectionPolicy. ResourcePoolTree.selectOneQueue(org.apache.drill.exec.ops.QueryContext, org.apache.drill.exec.resourcemgr.NodeResources) method is used by parallelizer to get a queue which will be used to admit a query. The selected queue resource constraints are used by parallelizer to allocate proper resources to a query so that it remains within the bounds.

The ResourcePools falls under 2 category:

Configurations Supported by ResourcePool:

A queue always have 1:1 relationship with a leaf pool. Queries are admitted and executed with a resource slice from the queue. It supports following configurations:

Once all the configuration are parsed an in-memory structures are created then for each query planner will select a queue where a query can be admitted. The queue selection process happens by traversing the ResourcePoolTree. During traversal process the query metadata is evaluated against assigned selector of a ResourcePool. If the selector returns true then traversal continues to it's child pools otherwise it stops there and tries another pool. With the traversal it finds all the leaf pools which are eligible for admitting the query and store that information in QueueAssignmentResult. Later the selected pools are passed to configured QueueSelectionPolicy to select one queue for the query. Planner uses that selected queue's max query memory per node parameter to limit resource assignment to all the fragments of a query on a node. After a query is planned with resource constraints it is sent to leader of that queue to ask for admission. If admitted the query required resources are reserved in global state store and query is executed on the cluster. For details please see the design document and functional spec linked in DRILL-7026
Skip navigation links

Copyright © 1970 The Apache Software Foundation. All rights reserved.