Package org.apache.drill.exec.store

Class PluginBootstrapLoaderImpl

java.lang.Object
org.apache.drill.exec.store.PluginBootstrapLoaderImpl
All Implemented Interfaces:
PluginBootstrapLoader
public class PluginBootstrapLoaderImpl extends Object implements PluginBootstrapLoader
Loads the set of bootstrap plugin configurations for new systems. Also handles upgrades to new and existing systems. Each set is given by a Jackson-serialized JSON file with the list of plugin (configurations).

The files are given via indirections in the config system to allow easier testing when "non-stock" configurations.

Bootstrap Files

When a new system starts (boots), Drill populates the persistent store with a "starter" (bootstrap) set of storage plugin configurations as provided by each connector locator. The classic locator uses one or more bootstrap files, serialized to JSON, and visible at the root of the class path. Each plugin implementation, if packaged in its own project, can provide a bootstrap file (bootstrap-storage-plugins.json).

The core plugins share a single file in the exec module which has a list of storage plugins used as default on fresh start up. Then we have contrib module where some plugins and formats reside. We want them to be included in default plugins as well during fresh start up which is done by reading each bootstrap file in the classpath.

Format Plugin Bootstrap

All works fine for the plugins, since each plugin also has a bootstrap-storage-plugins.json loaded via this process. However, format plugins are defined inside storage plugins. This presents a problem when the format is defined outside of the core (such as in the contrib module). In this case, when new format was added, the developer had to modify the bootstrap-storage-plugins.json in exec module to ensure that the new format was included.

The bootstrap-format-plugins.json solves this problem: it allows non-core modules to include formats during bootstrap. The file must still indicate which plugin (config) to modify, and must list the same format for each supported file system plugin such as dfs and {@ cp}. Despite this limitation, the additional file makes the non-core format plugins independent of the core files.

The resulting bootstrap process is to 1) gather the storage plugins, and 2) merge any format bootstrap into these plugins.

Upgrade Files

Bootstrap files populate a new system. There are times when a user upgrades their Drill install and we would like to push out configurations for newly added plugins. This is done via the optional upgrade file.

If the system is new, configurations from the upgrade file replace (override) values from the bootstrap file. The net effect, bootstrap plus upgrades, are written into the persistent store.

If the system is upgraded, then the current technique uses a highly unreliable system: an upgrade file exists. Any upgrades are applied on top of the user's stored values. The upgrade file is then deleted to avoid applying the same upgrades multiple times. Not idea, but it is what it is for now.

Format Plugins

Earlier versions of this code provided support for format plugins. However, this code never worked. The original design of Drill has format plugins as a set of attributes of the file system plugin config. Additional work is needed to allow file system configs to share plugins. Such work is a good idea, so the ill-fated format plugin code still appears, but does nothing.

Config Properties

ExecConstants.BOOTSTRAP_STORAGE_PLUGINS_FILE (drill.exec.storage.bootstrap.storage)
The name of the bootstrap file, normally bootstrap-storage-plugins.json.
ExecConstants.UPGRADE_STORAGE_PLUGINS_FILE (drill.exec.storage.upgrade.storage)
The name of the upgrade file, normally storage-plugins-override.conf. Unlike th bootstrap file, only one upgrade file can appear in the class path. The upgrade file is optional.