org.apache.drill.exec.store

Class PluginBootstrapLoaderImpl

java.lang.Object

org.apache.drill.exec.store.PluginBootstrapLoaderImpl

All Implemented Interfaces:

PluginBootstrapLoader

public class PluginBootstrapLoaderImpl
extends Object
implements PluginBootstrapLoader

Loads the set of bootstrap plugin configurations for new systems. Also handles upgrades to new and existing systems. Each set is given by a Jackson-serialized JSON file with the list of plugin (configurations).

The files are given via indirections in the config system to allow easier testing when "non-stock" configurations.

Bootstrap Files

When a new system starts (boots), Drill populates the persistent store with a "starter" (bootstrap) set of storage plugin configurations as provided by each connector locator. The classic locator uses one or more bootstrap files, serialized to JSON, and visible at the root of the class path. Each plugin implementation, if packaged in its own project, can provide a bootstrap file (bootstrap-storage-plugins.json).

The core plugins share a single file in the exec module which has a list of storage plugins used as default on fresh start up. Then we have contrib module where some plugins and formats reside. We want them to be included in default plugins as well during fresh start up which is done by reading each bootstrap file in the classpath.

Format Plugin Bootstrap

All works fine for the plugins, since each plugin also has a bootstrap-storage-plugins.json loaded via this process. However, format plugins are defined inside storage plugins. This presents a problem when the format is defined outside of the core (such as in the contrib module). In this case, when new format was added, the developer had to modify the bootstrap-storage-plugins.json in exec module to ensure that the new format was included.

The bootstrap-format-plugins.json solves this problem: it allows non-core modules to include formats during bootstrap. The file must still indicate which plugin (config) to modify, and must list the same format for each supported file system plugin such as dfs and . Despite this limitation, the additional file makes the non-core format plugins independent of the core files.

The resulting bootstrap process is to 1) gather the storage plugins, and 2) merge any format bootstrap into these plugins.

Upgrade Files

Bootstrap files populate a new system. There are times when a user upgrades their Drill install and we would like to push out configurations for newly added plugins. This is done via the optional upgrade file.

If the system is new, configurations from the upgrade file replace (override) values from the bootstrap file. The net effect, bootstrap plus upgrades, are written into the persistent store.

If the system is upgraded, then the current technique uses a highly unreliable system: an upgrade file exists. Any upgrades are applied on top of the user's stored values. The upgrade file is then deleted to avoid applying the same upgrades multiple times. Not idea, but it is what it is for now.

Format Plugins

Earlier versions of this code provided support for format plugins. However, this code never worked. The original design of Drill has format plugins as a set of attributes of the file system plugin config. Additional work is needed to allow file system configs to share plugins. Such work is a good idea, so the ill-fated format plugin code still appears, but does nothing.

Config Properties

ExecConstants.BOOTSTRAP_STORAGE_PLUGINS_FILE (drill.exec.storage.bootstrap.storage)

The name of the bootstrap file, normally bootstrap-storage-plugins.json.

ExecConstants.UPGRADE_STORAGE_PLUGINS_FILE (drill.exec.storage.upgrade.storage)

The name of the upgrade file, normally storage-plugins-override.conf. Unlike th bootstrap file, only one upgrade file can appear in the class path. The upgrade file is optional.

Constructor Summary

Constructors

Constructor and Description
PluginBootstrapLoaderImpl(PluginRegistryContext context)

Method Summary

Modifier and Type	Method and Description
StoragePlugins	bootstrapPlugins() Drill has detected that this is a new installation.
protected StoragePlugins	loadBootstrapPlugins(Map<String,URL> pluginURLMap)
void	onUpgrade() Drill successfully applied the plugin upgrades from PluginBootstrapLoader.updatedPlugins().
StoragePlugins	updatedPlugins() Get the new storage plugins from the ConfigConstants#STORAGE_PLUGINS_OVERRIDE_CONF file if it exists, null otherwise

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PluginBootstrapLoaderImpl

public PluginBootstrapLoaderImpl(PluginRegistryContext context)

Method Detail

bootstrapPlugins

public StoragePlugins bootstrapPlugins()
                                throws IOException

Description copied from interface: PluginBootstrapLoader

Drill has detected that this is a new installation. Provide the list of storage plugins (that is, names and configs) to use to populate the storage plugin store.

Specified by:

bootstrapPlugins in interface PluginBootstrapLoader

Throws:

IOException

loadBootstrapPlugins

protected StoragePlugins loadBootstrapPlugins(Map<String,URL> pluginURLMap)
                                       throws IOException

Throws:

IOException

updatedPlugins

public StoragePlugins updatedPlugins()

Get the new storage plugins from the ConfigConstants#STORAGE_PLUGINS_OVERRIDE_CONF file if it exists, null otherwise

Specified by:

updatedPlugins in interface PluginBootstrapLoader

onUpgrade

public void onUpgrade()

Description copied from interface: PluginBootstrapLoader

Drill successfully applied the plugin upgrades from PluginBootstrapLoader.updatedPlugins(). Use this event to mark this version as having been upgraded. (Again, this is crude and may not actually work. See DRILL-7613.)

Specified by:

onUpgrade in interface PluginBootstrapLoader