Package org.apache.drill.yarn.core

Class DrillOnYarnConfig

java.lang.Object
org.apache.drill.yarn.core.DrillOnYarnConfig
public class DrillOnYarnConfig extends Object
Configuration used within the Drill-on-YARN code. Configuration comes from four sources (in order of precedence):
  1. System properties
  2. $SITE_DIR/drill-on-yarn.conf
  3. Distribution-specific properties in $SITE_HOME/conf/doy-distrib.conf
  4. Drill-on-YARN defaults in drill-on-yarn-defaults.conf. (Which should be disjoint from the Drill settings.)
  5. Drill properties (via the Drill override system)

Defines constants for each property, including some defined in Drill. This provides a uniform property access interface even if some properties migrate between DoY and Drill proper.

  • Field Details

    • DEFAULTS_FILE_NAME

      public static final String DEFAULTS_FILE_NAME
      See Also:

    • DISTRIB_FILE_NAME

      public static final String DISTRIB_FILE_NAME
      See Also:

    • CONFIG_FILE_NAME

      public static final String CONFIG_FILE_NAME
      See Also:

    • DRILL_ON_YARN_PARENT

      public static final String DRILL_ON_YARN_PARENT
      See Also:

    • DOY_CLIENT_PARENT

      public static final String DOY_CLIENT_PARENT

    • DOY_AM_PARENT

      public static final String DOY_AM_PARENT

    • DOY_DRILLBIT_PARENT

      public static final String DOY_DRILLBIT_PARENT

    • FILES_PARENT

      public static final String FILES_PARENT

    • DFS_PARENT

      public static final String DFS_PARENT

    • HTTP_PARENT

      public static final String HTTP_PARENT

    • YARN_PARENT

      public static final String YARN_PARENT

    • HADOOP_PARENT

      public static final String HADOOP_PARENT

    • CLIENT_PARENT

      public static final String CLIENT_PARENT

    • APP_NAME

      public static final String APP_NAME

    • CLUSTER_ID

      public static final String CLUSTER_ID
      See Also:

    • DFS_CONNECTION

      public static final String DFS_CONNECTION

    • DFS_APP_DIR

      public static final String DFS_APP_DIR

    • YARN_QUEUE

      public static final String YARN_QUEUE

    • YARN_PRIORITY

      public static final String YARN_PRIORITY

    • DRILL_ARCHIVE_PATH

      public static final String DRILL_ARCHIVE_PATH

    • DRILL_DIR_NAME

      public static final String DRILL_DIR_NAME

    • DRILL_ARCHIVE_KEY

      public static final String DRILL_ARCHIVE_KEY
      Key used for the Drill archive file in the AM launch config.

    • SITE_ARCHIVE_KEY

      public static final String SITE_ARCHIVE_KEY

    • LOCALIZE_DRILL

      public static final String LOCALIZE_DRILL

    • CONF_AS_SITE

      public static final String CONF_AS_SITE

    • DRILL_HOME

      public static final String DRILL_HOME

    • SITE_DIR

      public static final String SITE_DIR

    • JAVA_LIB_PATH

      public static final String JAVA_LIB_PATH

    • HADOOP_HOME

      public static final String HADOOP_HOME

    • HADOOP_CLASSPATH

      public static final String HADOOP_CLASSPATH

    • HBASE_CLASSPATH

      public static final String HBASE_CLASSPATH

    • MEMORY_KEY

      public static final String MEMORY_KEY
      See Also:

    • VCORES_KEY

      public static final String VCORES_KEY
      See Also:

    • DISKS_KEY

      public static final String DISKS_KEY
      See Also:

    • VM_ARGS_KEY

      public static final String VM_ARGS_KEY
      See Also:

    • HEAP_KEY

      public static final String HEAP_KEY
      See Also:

    • AM_MEMORY

      public static final String AM_MEMORY

    • AM_VCORES

      public static final String AM_VCORES

    • AM_DISKS

      public static final String AM_DISKS

    • AM_NODE_LABEL_EXPR

      public static final String AM_NODE_LABEL_EXPR

    • AM_HEAP

      public static final String AM_HEAP

    • AM_VM_ARGS

      public static final String AM_VM_ARGS

    • AM_POLL_PERIOD_MS

      public static final String AM_POLL_PERIOD_MS

    • AM_TICK_PERIOD_MS

      public static final String AM_TICK_PERIOD_MS

    • AM_PREFIX_CLASSPATH

      public static final String AM_PREFIX_CLASSPATH

    • AM_CLASSPATH

      public static final String AM_CLASSPATH

    • AM_DEBUG_LAUNCH

      public static final String AM_DEBUG_LAUNCH

    • AM_ENABLE_AUTO_SHUTDOWN

      public static final String AM_ENABLE_AUTO_SHUTDOWN

    • DRILLBIT_MEMORY

      public static final String DRILLBIT_MEMORY

    • DRILLBIT_VCORES

      public static final String DRILLBIT_VCORES

    • DRILLBIT_DISKS

      public static final String DRILLBIT_DISKS

    • DRILLBIT_VM_ARGS

      public static final String DRILLBIT_VM_ARGS

    • DRILLBIT_HEAP

      public static final String DRILLBIT_HEAP

    • DRILLBIT_DIRECT_MEM

      public static final String DRILLBIT_DIRECT_MEM

    • DRILLBIT_CODE_CACHE

      public static final String DRILLBIT_CODE_CACHE

    • DRILLBIT_LOG_GC

      public static final String DRILLBIT_LOG_GC

    • DRILLBIT_PREFIX_CLASSPATH

      public static final String DRILLBIT_PREFIX_CLASSPATH

    • DRILLBIT_EXTN_CLASSPATH

      public static final String DRILLBIT_EXTN_CLASSPATH

    • DRILLBIT_CLASSPATH

      public static final String DRILLBIT_CLASSPATH

    • DRILLBIT_MAX_RETRIES

      public static final String DRILLBIT_MAX_RETRIES

    • DRILLBIT_DEBUG_LAUNCH

      public static final String DRILLBIT_DEBUG_LAUNCH

    • DRILLBIT_HTTP_PORT

      public static final String DRILLBIT_HTTP_PORT
      See Also:

    • DISABLE_YARN_LOGS

      public static final String DISABLE_YARN_LOGS

    • DRILLBIT_USER_PORT

      public static final String DRILLBIT_USER_PORT
      See Also:

    • DRILLBIT_BIT_PORT

      public static final String DRILLBIT_BIT_PORT
      See Also:

    • DRILLBIT_USE_HTTPS

      public static final String DRILLBIT_USE_HTTPS
      See Also:

    • DRILLBIT_MAX_EXTRA_NODES

      public static final String DRILLBIT_MAX_EXTRA_NODES

    • DRILLBIT_REQUEST_TIMEOUT_SEC

      public static final String DRILLBIT_REQUEST_TIMEOUT_SEC

    • ZK_CONNECT

      public static final String ZK_CONNECT
      See Also:

    • ZK_ROOT

      public static final String ZK_ROOT
      See Also:

    • ZK_FAILURE_TIMEOUT_MS

      public static final String ZK_FAILURE_TIMEOUT_MS
      See Also:

    • ZK_RETRY_COUNT

      public static final String ZK_RETRY_COUNT
      See Also:

    • ZK_RETRY_DELAY_MS

      public static final String ZK_RETRY_DELAY_MS
      See Also:

    • HTTP_ENABLED

      public static final String HTTP_ENABLED

    • HTTP_ENABLE_SSL

      public static final String HTTP_ENABLE_SSL

    • HTTP_PORT

      public static final String HTTP_PORT

    • HTTP_AUTH_TYPE

      public static final String HTTP_AUTH_TYPE

    • HTTP_REST_KEY

      public static final String HTTP_REST_KEY

    • HTTP_SESSION_MAX_IDLE_SECS

      public static final String HTTP_SESSION_MAX_IDLE_SECS

    • HTTP_REFRESH_SECS

      public static final String HTTP_REFRESH_SECS

    • HTTP_USER_NAME

      public static final String HTTP_USER_NAME

    • HTTP_PASSWORD

      public static final String HTTP_PASSWORD

    • AUTH_TYPE_NONE

      public static final String AUTH_TYPE_NONE
      See Also:

    • AUTH_TYPE_DRILL

      public static final String AUTH_TYPE_DRILL
      See Also:

    • AUTH_TYPE_SIMPLE

      public static final String AUTH_TYPE_SIMPLE
      See Also:

    • CLIENT_POLL_SEC

      public static final String CLIENT_POLL_SEC

    • CLIENT_START_WAIT_SEC

      public static final String CLIENT_START_WAIT_SEC

    • CLIENT_STOP_WAIT_SEC

      public static final String CLIENT_STOP_WAIT_SEC

    • CLUSTERS

      public static final String CLUSTERS

    • LOCAL_DIR_NAME

      public static String LOCAL_DIR_NAME
      Name of the subdirectory of the container directory that will hold localized Drill distribution files. This name must be consistent between AM launch request and AM launch, and between Drillbit launch request and Drillbit launch. This name is fixed; there is no reason for the user to change it as it is visible only in the YARN container environment.

    • APP_ID_ENV_VAR

      public static final String APP_ID_ENV_VAR
      See Also:

    • DRILL_ARCHIVE_ENV_VAR

      public static final String DRILL_ARCHIVE_ENV_VAR
      See Also:

    • SITE_ARCHIVE_ENV_VAR

      public static final String SITE_ARCHIVE_ENV_VAR
      See Also:

    • DRILL_HOME_ENV_VAR

      public static final String DRILL_HOME_ENV_VAR
      See Also:

    • DRILL_SITE_ENV_VAR

      public static final String DRILL_SITE_ENV_VAR
      See Also:

    • AM_HEAP_ENV_VAR

      public static final String AM_HEAP_ENV_VAR
      See Also:

    • AM_JAVA_OPTS_ENV_VAR

      public static final String AM_JAVA_OPTS_ENV_VAR
      See Also:

    • DRILL_CLASSPATH_ENV_VAR

      public static final String DRILL_CLASSPATH_ENV_VAR
      See Also:

    • DRILL_CLASSPATH_PREFIX_ENV_VAR

      public static final String DRILL_CLASSPATH_PREFIX_ENV_VAR
      See Also:

    • DOY_LIBPATH_ENV_VAR

      public static final String DOY_LIBPATH_ENV_VAR
      See Also:

    • DRILL_DEBUG_ENV_VAR

      public static final String DRILL_DEBUG_ENV_VAR
      See Also:

    • SITE_ARCHIVE_NAME

      public static final String SITE_ARCHIVE_NAME
      The name of the Drill site archive stored in dfs. Since the archive is created by the client as a temp file, it's local name has no meaning; we use this standard name on dfs.
      See Also:

    • instance

      protected static DrillOnYarnConfig instance

  • Constructor Details

    • DrillOnYarnConfig

      protected DrillOnYarnConfig()

  • Method Details

    • append

      public static String append(String parent, String key)

    • load

      public static DrillOnYarnConfig load() throws DoyConfigException
      Throws:
      DoyConfigException

    • doLoad

      protected void doLoad(ClassLoader cl) throws DoyConfigException
      Load the config.
      Parameters:
      cl - class loader to use for resource searches (except defaults). Allows test to specify a specialized version.

      Implemented in a way that allows unit testing. The parseUrl( ) methods let us mock the files; the load( ) methods seem to not actually use the provided class loader.

      Throws:
      DoyConfigException

    • getDrillConfig

      public DrillConfig getDrillConfig()

    • getClassPathScan

      public ScanResult getClassPathScan()
      Return Drill's class path scan. This is used only in the main thread during initialization. Not needed by the client, so done in an unsynchronized, lazy fashion.
      Returns:
      Drill's class path scan

    • setClientPaths

      public void setClientPaths() throws DoyConfigException
      Obtain Drill home from the DRILL_HOME environment variable set by drill-config.sh, which is called from drill-on-yarn.sh. When debugging, DRILL_HOME must be set in the environment.

      This information is required only by the client to prepare for uploads to DFS.

      Throws:
      DoyConfigException

    • getEnv

      protected String getEnv(String key)
      All environment variable access goes through this function to allow unit tests to replace this function to set test values. (The Java environment is immutable, so it is not possible for unit tests to change the actual environment.)
      Parameters:
      key - key to allow unit tests to replace this function
      Returns:
      environment variable

    • setAmDrillHome

      public void setAmDrillHome() throws DoyConfigException
      Retrieve the AM Drill home location from the DRILL_HOME variable set in the drill-am.sh launch script.
      Throws:
      DoyConfigException

    • getConfig

      public com.typesafe.config.Config getConfig()

    • instance

      public static DrillOnYarnConfig instance()

    • config

      public static com.typesafe.config.Config config()

    • getLocalDrillHome

      public File getLocalDrillHome()
      Return the Drill home on this machine as inferred from the config file contents or location.
      Returns:
      Drill home

    • dump

      public void dump()

    • dumpEnv

      public void dumpEnv(PrintStream out)

    • getPairs

      public List<NameValuePair> getPairs()

    • clusterGroupKey

      public static String clusterGroupKey(int index, String key)

    • clusterGroupCount

      public int clusterGroupCount()

    • findSuffix

      public static String findSuffix(String baseName)

    • getRemoteDrillHome

      public String getRemoteDrillHome() throws DoyConfigException
      Get the location of Drill home on a remote machine, relative to the container working directory. Used when constructing a launch context. Assumes either the absolute path from the config file, or a constructed path to the localized Drill on the remote node. YARN examples use "./foo" to refer to container resources. But, since we cannot be sure when such a path is evaluated, we explicitly use YARN's PWD environment variable to get the absolute path.
      Returns:
      the remote path, with the "$PWD" environment variable.
      Throws:
      DoyConfigException

    • getRemoteSiteDir

      public String getRemoteSiteDir()
      Get the optional remote site directory name. This name will include the absolute path for a non-localized application. It will return the path relative to the container for a localized application. In the localized case, the site archive is tar'ed relative to the site directory so that its contents are unarchived directly into the YARN-provided folder (with the name of the archive) key. That is, if the site directory on the client is /var/drill/my-site, the contents of the tar file will be "./drill-override.conf", etc., and the remote location is $PWD/site-key/drill-override.conf, where site-key is the key name used to localize the site archive.
      Returns:
      remote site directory name

    • getLocalAppIdFile

      public File getLocalAppIdFile()
      Return the app ID file to use for this client run. The file is in the directory that holds the site dir (if a site dir is used), else the directory that holds Drill home (otherwise.) Not that the file does NOT go into the site dir or Drill home as we upload these directories (via archives) to DFS so we don't want to change them by adding a file.

      It turns out that Drill allows two distinct clusters to share the same ZK root and/or cluster ID (just not the same combination), so the file name contains both parts.

      Returns:
      local app id file

    • hasSiteDir

      public boolean hasSiteDir()

    • getLocalSiteDir

      public File getLocalSiteDir()

    • getDrillArchiveDfsPath

      public String getDrillArchiveDfsPath()
      Returns the DFS path to the localized Drill archive. This is an AM-only method as it relies on an environment variable set by the client. It is set only if the application is localized, it is not set for a non-localized run.
      Returns:
      the DFS path to the localized Drill archive

    • getSiteArchiveDfsPath

      public String getSiteArchiveDfsPath()
      Returns the DFS path to the localized site archive. This is an AM-only method as it relies on an environment variable set by the client. This variable is optional; if not set then the AM can infer that the application does not use a site archive (configuration files reside in $DRILL_HOME/conf), or the application is not localized.
      Returns:
      the DFS path to the localized site archive