Package org.apache.drill.exec.work.foreman.rm

Class DistributedQueryQueue

java.lang.Object
org.apache.drill.exec.work.foreman.rm.DistributedQueryQueue
All Implemented Interfaces:
QueryQueue
public class DistributedQueryQueue extends Object implements QueryQueue
Distributed query queue which uses a Zookeeper distributed semaphore to control queuing across the cluster. The distributed queue is actually two queues: one for "small" queries, another for "large" queries. Query size is determined by the Planner's estimate of query cost.

This queue is configured using system options:

exec.queue.enable
Set to true to enable the distributed queue.
exec.queue.large
The maximum number of large queries to admit. Additional queries wait in the queue.
exec.queue.small
The maximum number of small queries to admit. Additional queries wait in the queue.
exec.queue.threshold
The cost threshold. Queries below this size are small, at or above this size are large..
exec.queue.timeout_millis
The maximum time (in milliseconds) a query will wait in the queue before failing.

The above values are refreshed every five seconds. This aids performance a bit in systems with very high query arrival rates.

  • Constructor Details

  • Method Details

    • setMemoryPerNode

      public void setMemoryPerNode(long memoryPerNode)
      Specified by:
      setMemoryPerNode in interface QueryQueue

    • defaultQueryMemoryPerNode

      public long defaultQueryMemoryPerNode(double cost)
      Description copied from interface: QueryQueue
      Return the amount of memory per node when creating a EXPLAIN query plan. Plans to be executed should get the query memory from the lease, as the lease may adjust the default amount on a per-query basis. This means that the memory used to execute the query may differ from the amount shown in an EXPLAIN plan.
      Specified by:
      defaultQueryMemoryPerNode in interface QueryQueue
      Returns:
      assumed memory per node, in bytes, to use when creating an EXPLAIN plan

    • minimumOperatorMemory

      public long minimumOperatorMemory()
      Description copied from interface: QueryQueue
      Optional floor on the amount of memory assigned per operator. This ensures that operators receive a certain amount, separate from any memory slicing. This can oversubscribe node memory if used incorrectly.
      Specified by:
      minimumOperatorMemory in interface QueryQueue
      Returns:
      minimum per-operator memory, in bytes

    • enqueue

      public QueryQueue.QueueLease enqueue(UserBitShared.QueryId queryId, double cost) throws QueryQueue.QueryQueueException, QueryQueue.QueueTimeoutException
      This limits the number of "small" and "large" queries that a Drill cluster will run simultaneously, if queuing is enabled. If the query is unable to run, this will block until it can. Beware that this is called under run(), and so will consume a thread while it waits for the required distributed semaphore.
      Specified by:
      enqueue in interface QueryQueue
      Parameters:
      queryId - query identifier
      cost - the query plan
      Returns:
      the query lease which must be passed to #release(QueueLease) upon query completion
      Throws:
      QueryQueue.QueryQueueException - if the underlying ZK queuing mechanism fails
      QueryQueue.QueueTimeoutException - if the query waits too long in the queue

    • enabled

      public boolean enabled()
      Description copied from interface: QueryQueue
      Determine if the queue is enabled.
      Specified by:
      enabled in interface QueryQueue
      Returns:
      true if the query is enabled, false otherwise.

    • getInfo

      public DistributedQueryQueue.ZKQueueInfo getInfo()

    • close

      public void close()
      Specified by:
      close in interface QueryQueue