Package org.apache.drill.yarn.client

Class DrillOnYarn

java.lang.Object
org.apache.drill.yarn.client.DrillOnYarn
public class DrillOnYarn extends Object
Client for the Drill-on-YARN integration. See YARN documentation for the role of a YARN client.

The client needs configuration information from drill-on-yarn.conf, the directory of which must be in the class path. It is put there by the drill-on-yarn.sh script.

The client also requires a debugging configuration file to be given explicitly as follows:
-Dlogback.configurationFile=/path/to/yarn-client-log.xml
The drillbit itself uses the default logging config file name of logback.xml; which contains references to system properties that are not defined in this client. The result of not including the log configuration file is that you'll see various var.name_IS_UNDEFINED files in the directory from which you launched the client.

The client accepts a command, creates a command object for that command, and executes it. There are a few main commands (start, stop), along with some management commands (status, resize), and a few commands mostly used for debugging and diagnosis (upload,etc.) Some commands are very similar, so a single command object may handle multiple user commands.

The client requires a working distributed file system (DFS), the configuration of which is given either implicitly, or in the Hadoop configuration files. Similarly, the client requires a working YARN deployment, again with either implicit configuration or configuration given in the Hadoop configuration. The Hadoop configuration must be on the class path when launching this client.

Debugging

To debug this class, add two or three directories to your class path:

  • $DRILL_CONF_DIR (if using a separate site directory)
  • $HADOOP_HOME/etc/hadoop
  • $DRILL_HOME/conf
Note that these MUST be in the order listed since $DRILL_HOME/conf contains, by default, a version of core-site.xml that probably is NOT the one you want to use for YARN. For YARN, you want the one in $HADOOP_HOME/etc/hadoop.

Also, set the following VM argument:
-Dlogback.configurationFile=/path/to/drill/conf/yarn-client-log.xml
or
-Dlogback.configurationFile=/path/to/drill-site/yarn-client-log.xml

  • Constructor Details

    • DrillOnYarn

      public DrillOnYarn()

  • Method Details

    • main

      public static void main(String[] argv)

    • run

      public static void run(String[] argv)