Package org.apache.drill.yarn.appMaster

Class SchedulerStateImpl

java.lang.Object
org.apache.drill.yarn.appMaster.SchedulerStateImpl
All Implemented Interfaces:
SchedulerState, SchedulerStateActions
public final class SchedulerStateImpl extends Object implements SchedulerState, SchedulerStateActions
Manages a the set of tasks associated with a scheduler. The scheduler decides which tasks to run or stop; the task group manages the life-cycle of the tasks for the given scheduler.

Schedulers, and hence their groups, define a priority. When starting, higher priority (lower priority value) groups run before lower priority groups. Similarly, when shrinking the cluster, lower priority groups shrink before higher priority groups.

  • Field Details

    • pendingTasks

      protected List<Task> pendingTasks
      Tracks the tasks to be started, but for which no work has yet been done. (State == PENDING).

    • allocatingTasks

      protected List<Task> allocatingTasks
      Tracks the tasks for which containers have been requested. (State == REQUESTED).

    • activeContainers

      protected Map<org.apache.hadoop.yarn.api.records.ContainerId,Task> activeContainers
      Tracks running tasks: those that have been allocated containers and are starting, running, failed or ended. We use a map for this because, during these states, the task is identified by its container. (State == LAUNCHING, RUNNING or ENDING).

  • Constructor Details

  • Method Details

    • getName

      public String getName()
      Description copied from interface: SchedulerStateActions
      Returns the name of the scheduler associated with this task action group.
      Specified by:
      getName in interface SchedulerStateActions
      Returns:
      The name of the scheduler associated with this task action group.

    • getMaxRetries

      public int getMaxRetries()

    • getStopTimeoutMs

      public int getStopTimeoutMs()

    • getScheduler

      public Scheduler getScheduler()
      Description copied from interface: SchedulerStateActions
      Returns the scheduler associated with this task group.
      Specified by:
      getScheduler in interface SchedulerStateActions
      Returns:
      The scheduler associated with this task group.

    • start

      public void start(Task task)
      Define a new task in this group. Adds it to the pending queue so that a container will be requested.
      Specified by:
      start in interface SchedulerState
      Parameters:
      task -

    • enqueuePendingRequest

      public void enqueuePendingRequest(Task task)
      Put a task into the queue waiting to send a container request to YARN.
      Parameters:
      task -

    • maxCurrentRequests

      public int maxCurrentRequests()

    • requestContainers

      public boolean requestContainers(EventContext context, int maxRequests)
      Description copied from interface: SchedulerStateActions
      Request a container the first task that we wish to start.
      Specified by:
      requestContainers in interface SchedulerStateActions

    • dequeuePendingRequest

      public void dequeuePendingRequest(Task task)
      Remove a task from the queue of tasks waiting to send a container request. The caller must put the task into the proper next state: the allocating queue or the completed task list.
      Parameters:
      task -

    • enqueueAllocatingTask

      public void enqueueAllocatingTask(Task task)
      Put a task onto the queue awaiting an allocation response from YARN.
      Parameters:
      task -

    • containerAllocated

      public void containerAllocated(EventContext context, org.apache.hadoop.yarn.api.records.Container container)
      Description copied from interface: SchedulerStateActions
      A container request has been granted. Match the container up with the first task waiting for a container and launch the task.
      Specified by:
      containerAllocated in interface SchedulerStateActions

    • checkTasks

      public void checkTasks(EventContext context, long curTime)
      Specified by:
      checkTasks in interface SchedulerStateActions

    • dequeueAllocatingTask

      public void dequeueAllocatingTask(Task task)
      Remove a task from the list of those waiting for a container allocation. The allocation may be done, or cancelled. The caller is responsible for moving the task to the next collection.
      Parameters:
      task -

    • containerAllocated

      public void containerAllocated(Task task)
      Mark that a task has become active and should be tracked by its container ID. Prior to this, the task is not associated with a container.
      Parameters:
      task -

    • containerReleased

      public void containerReleased(Task task)
      Mark that a task has completed: its container has expired or been revoked or the task has completed: successfully or a failure, as given by the task's disposition. The task can no longer be tracked by its container ID. If this is the last active task for this group, mark the group itself as completed.
      Parameters:
      task -

    • taskEnded

      public void taskEnded(Task task)
      Mark that a task has completed successfully or a failure, as given by the task's disposition. If this is the last active task for this group, mark the group itself as completed.
      Parameters:
      task -

    • taskRetried

      public void taskRetried(Task task)
      Mark that a task is about to be retried. Task still retains its state from the current try.
      Parameters:
      task -

    • shutDown

      public void shutDown(EventContext context)
      Description copied from interface: SchedulerStateActions
      Shut down this task group by canceling all tasks not already cancelled.
      Specified by:
      shutDown in interface SchedulerStateActions

    • hasTasks

      public boolean hasTasks()
      Report if this task group has any tasks in the active part of their life-cycle: pending, allocating or active.
      Returns:

    • isDone

      public boolean isDone()
      Description copied from interface: SchedulerStateActions
      Determine if this task group is done. It is done when there are no active tasks and the controller itself is shutting down. This latter check differentiates the start state (when no tasks are active) from the end state. The AM will not shut down until all task groups are done.
      Specified by:
      isDone in interface SchedulerStateActions
      Returns:
      True if this task group is done. False otherwise.

    • adjustTasks

      public void adjustTasks()
      Description copied from interface: SchedulerStateActions
      Adjust the number of running tasks as needed to balance the number of running tasks with the desired number. May result in no change it the cluster is already in balance (or is in the process of achieving balance.)
      Specified by:
      adjustTasks in interface SchedulerStateActions

    • requestStop

      public boolean requestStop(Task task)
      Request a graceful stop of the task. Delegates to the task manager to do the actual work.
      Returns:
      true if the graceful stop request was sent, false if not, or if this task type has no graceful stop

    • getTaskCount

      public int getTaskCount()
      Description copied from interface: SchedulerState
      The number of tasks in any active (non-ended) lifecycle state.
      Specified by:
      getTaskCount in interface SchedulerState
      Specified by:
      getTaskCount in interface SchedulerStateActions
      Returns:
      The number of tasks in any active (non-ended) lifecycle state.

    • getCancelledTaskCount

      public int getCancelledTaskCount()
      Description copied from interface: SchedulerState
      The number of active tasks that have been cancelled, but have not yet ended.
      Specified by:
      getCancelledTaskCount in interface SchedulerState

    • getStartingTasks

      public List<Task> getStartingTasks()
      Description copied from interface: SchedulerState
      Returns the list of tasks awaiting a container request to be sent to YARN or for which a container request has been sent to YARN, but no container allocation has yet been received. Such tasks are simple to cancel. The list does not contain any tasks in this state which have previously been cancelled.
      Specified by:
      getStartingTasks in interface SchedulerState
      Returns:
      The list of tasks awaiting a container request to be sent to YARN or for which a container request has been sent to YARN, but no container allocation has yet been received.

    • getActiveTasks

      public List<Task> getActiveTasks()
      Description copied from interface: SchedulerState
      Returns the list of active tasks that have not yet been cancelled. Active tasks are any task for which a container has been assigned, but has not yet received a RM container completion event.
      Specified by:
      getActiveTasks in interface SchedulerState
      Returns:
      The list of active tasks that have not yet been cancelled.

    • cancel

      public void cancel(Task task)
      Specified by:
      cancel in interface SchedulerState
      Specified by:
      cancel in interface SchedulerStateActions

    • getLiveCount

      public int getLiveCount()
      Specified by:
      getLiveCount in interface SchedulerStateActions

    • visitTaskModels

      public void visitTaskModels(TaskVisitor visitor)
      Specified by:
      visitTaskModels in interface SchedulerStateActions

    • getTask

      public Task getTask(int id)
      Specified by:
      getTask in interface SchedulerStateActions

    • getRequestCount

      public int getRequestCount()
      Specified by:
      getRequestCount in interface SchedulerStateActions

    • getController

      public ClusterController getController()
      Specified by:
      getController in interface SchedulerState