StoragePluginRegistryImpl (Drill : 1.20.3 API)

java.lang.Object
- org.apache.drill.exec.store.StoragePluginRegistryImpl

All Implemented Interfaces:

AutoCloseable, Iterable<Map.Entry<String,StoragePlugin>>, StoragePluginRegistry
```
public class StoragePluginRegistryImpl
extends Object
implements StoragePluginRegistry
```
Plugin registry. Caches plugin instances which correspond to configurations stored in persistent storage. Synchronizes the instances and storage.
Allows multiple "locators" to provide plugin classes such as the "classic" version for classes in the same class loader, the "system" version for system-defined plugins.
provides multiple layers of abstraction:
- A plugin config/implementation pair (called a "connector" here) is located by
- A connector locator, which also provides bootstrap plugins and can create a plugin instance from a configuration, which are cached in
- The plugin cache, which holds stored, system and ad-hoc plugins. The stored plugins are backed by
- A persistent store: the file system for tests and embedded, ZK for a distibuted server, or
- An ephemeral cache for unnamed configs, such as those created by a table function.
The idea is to push most functionality into the above abstractions, leaving overall coordination here.
Plugins themselves have multiple levels of definitions:
- The config and plugin classes, provided by the locator.
- The ConnectorHandle which defines the config class and the locator which can create instances of that class.
- A config instance which is typically deserialized from JSON independent of the implementation class.
- A PluginHandle which pairs the config with a name as the unit that the user thinks of as a "plugin." The plugin entry links to the ConnectorEntry to create the instance lazily when first requested.
- The plugin class instance, which provides long-term state and which provides the logic for the plugin.
Concurrency
Drill is a concurrent system; multiple users can attempt to add, remove and update plugin configurations at the same time. The only good solution would be to version the plugin configs. Instead, we rely on the fact that configs change infrequently.
The code syncs the in-memory cache with the persistent store on each access (which is actually inefficient and should be reviewed.)
During refresh, it could be that another thread is doing exactly the same thing, or even fighting us by changing the config. It is impossible to ensure a totally consistent answer. The goal is to make sure that the cache ends up agreeing with the persistent store as it was at some point in time.
The PluginsMap class provides in-memory synchronization of the name and config maps. Careful coding is needed when handling refresh since another thread could make the same changes.
Once the planner obtains a plugin, another user could come along and change the config for that plugin. Drill treats that change as another plugin: the original one continues to be used by the planner (but see below), while new queries use the new version.
Since the config on remote servers may have changed relative to the one this Foreman used for planning, the plan includes the plugin config itself (not just a reference to the config.) This works because the config is usually small.
Ephemeral Plugins
An ephemeral plugin handles table functions which create a temporary, unnamed configuration that is needed only for the execution of a single query, but which may be used across many threads. If the same table function is used multiple times, then the same ephemeral plugin will be used across queries. Ephemeral plugins are are based on the same connectors as stored plugins, but are not visible to the planner. They will expire after some time or number.
The ephemeral store also acts as a graveyard for deleted or changed plugins. When removing a plugin, the old plugin is moved to ephemeral storage to allow running queries to locate it. Similarly, when a new configuration is stored, the corresponding plugin is retrieved from ephemeral storage, if it exists. This avoids odd cases where the same plugin exists in both normal and ephemeral storage.
Caveats
The main problem with synchronization at present is that plugins provide a close() method that, if used, could render the plugin unusable. Suppose a Cassandra plugin, say, maintains a connection to a server used across multiple queries and threads. Any change to the config immediately calls close() on the plugin, even though it may be in use in planning a query on another thread. Random failures will result.
The same issue can affect ephemeral plugins: if the number in the cache reaches the limit, the registry will start closing old ones, without knowning if that plugin is actually in use.
The workaround is to not actually honor the close() call. Longer term, a reference count is needed.
Error Handling
Error handling needs review. Those problems that result from user actions should be raised as a UserException. Those that violate invariants as other forms of exception.

Nested Class Summary
- Nested classes/interfaces inherited from interface org.apache.drill.exec.store.StoragePluginRegistry
  StoragePluginRegistry.PluginEncodingException, StoragePluginRegistry.PluginException, StoragePluginRegistry.PluginFilter, StoragePluginRegistry.PluginNotFoundException

Field Summary
- Fields inherited from interface org.apache.drill.exec.store.StoragePluginRegistry
  PSTORE_NAME

Constructor Summary

Constructors
Constructor and Description

StoragePluginRegistryImpl(DrillbitContext context)

Constructors
Constructor and Description
`StoragePluginRegistryImpl(DrillbitContext context)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`Set<String>`	`availablePlugins()` Returns the set of available plugin names.
`void`	`close()`
`StoragePluginConfig`	`copyConfig(StoragePluginConfig orig)` Copy the given storage plugin config so it may be modified.
`StoragePluginConfig`	`copyConfig(String name)` Copy a stored config so that it can be modified.
`protected static void`	`copyPluginStatus(StoragePluginConfig oldPluginConfig, StoragePluginConfig newPluginConfig)` Identifies the enabled status for new storage plugins config.
`StoragePluginConfig`	`decode(String json)` Return a config decoded from JSON.
`Map<String,StoragePluginConfig>`	`enabledConfigs()` Returns a copy of the set of enabled stored plugin configurations.
`String`	`encode(StoragePluginConfig config)`
`String`	`encode(String name)` Return a config encoded as JSON.
`StoragePluginConfig`	`getDefinedConfig(String name)` Retrieve an available configuration.
`FormatPlugin`	`getFormatPlugin(StoragePluginConfig storageConfig, FormatPluginConfig formatConfig)`
`FormatPlugin`	`getFormatPluginByConfig(StoragePluginConfig storageConfig, FormatPluginConfig formatConfig)` Get the Format plugin for the FileSystemPlugin associated with the provided storage config and format config.
`StoragePlugin`	`getPlugin(StoragePluginConfig config)`
`StoragePlugin`	`getPlugin(String name)` Get a plugin by name.
`StoragePlugin`	`getPluginByConfig(StoragePluginConfig config)` Get a plugin by configuration.
`SchemaFactory`	`getSchemaFactory()` Get the Schema factory associated with this storage plugin registry.
`StoragePluginConfig`	`getStoredConfig(String name)` Configs are obtained from the persistent store.
`void`	`init()` Initialize the storage plugin registry.
`Iterator<Map.Entry<String,StoragePlugin>>`	`iterator()`
`com.fasterxml.jackson.databind.ObjectMapper`	`mapper()` Object mapper to read/write the JSON form of a plugin.
`void`	`put(String name, StoragePluginConfig config)` Store a plugin by name and configuration.
`void`	`putFormatPlugin(String pluginName, String formatName, FormatPluginConfig formatConfig)` Safe way to add or remove a format plugin config from a stored file system configuration.
`void`	`putJson(String name, String json)` Put a storage plugin config from JSON.
`void`	`remove(String name)` Remove a plugin by name
`<T extends StoragePlugin> T`	`resolve(StoragePluginConfig storageConfig, Class<T> desired)`
`<T extends FormatPlugin> T`	`resolveFormat(StoragePluginConfig storageConfig, FormatPluginConfig formatConfig, Class<T> desired)` Resolve a storage plugin given a storage plugin config.
`void`	`setEnabled(String name, boolean enable)` Set the plugin to the requested enabled state.
`Map<String,StoragePluginConfig>`	`storedConfigs()` Returns a set of all stored plugin configurations, directly from the persistent store.
`Map<String,StoragePluginConfig>`	`storedConfigs(StoragePluginRegistry.PluginFilter filter)` Return a possibly-filtered set of plugins from the persistent store.
`void`	`validatedPut(String name, StoragePluginConfig config)` Like `StoragePluginRegistry.put(String, StoragePluginConfig)`, but forces instantiation of the plugin to verify that the configuration is valid at this moment in time.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable
forEach, spliterator

- Constructor Detail
  - StoragePluginRegistryImpl
```
public StoragePluginRegistryImpl(DrillbitContext context)
```
- Method Detail
  - init
```
public void init()
```
    Description copied from interface: StoragePluginRegistry
    
    Initialize the storage plugin registry. Must be called before the registry is used.
    
    Specified by:
    
    init in interface StoragePluginRegistry
  - copyPluginStatus
```
protected static void copyPluginStatus(StoragePluginConfig oldPluginConfig,
                                       StoragePluginConfig newPluginConfig)
```
    Identifies the enabled status for new storage plugins config. If this status is absent in the updater file, the status is kept from the configs, which are going to be updated
    
    Parameters:
    
    oldPluginConfig - current storage plugin config from Persistent Store or bootstrap config file
    
    newPluginConfig - new storage plugin config
  - put
```
public void put(String name,
                StoragePluginConfig config)
         throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Store a plugin by name and configuration. If the plugin already exists, update the plugin. This form directly updates persistent storage. The in-memory cache is updated on the next refresh. This form will accept an invalid plugin, which will be disabled upon refresh. Since Drill is distributed, and plugins work with external systems, the external system can become invalid at any moment (not just when pugins are updated), so the model used for put() mimics normal runtime operation.
    
    Specified by:
    
    put in interface StoragePluginRegistry
    
    Parameters:
    
    name - The name of the plugin
    
    config - The plugin configuration
    
    Throws:
    
    StoragePluginRegistry.PluginException - if plugin cannot be created
  - validatedPut
```
public void validatedPut(String name,
                         StoragePluginConfig config)
                  throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Like StoragePluginRegistry.put(String, StoragePluginConfig), but forces instantiation of the plugin to verify that the configuration is valid at this moment in time.
    
    Specified by:
    
    validatedPut in interface StoragePluginRegistry
    
    Throws:
    
    StoragePluginRegistry.PluginException
  - setEnabled
```
public void setEnabled(String name,
                       boolean enable)
                throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Set the plugin to the requested enabled state. Does nothing if the plugin is already in the requested state. If a formerly enabled plugin is disabled, moves the plugin from the in-memory cache to the ephemeral store. If a formerly disabled plugin is enabled, verifies that the plugin can be instantiated as for #verifiedPut().
    Use this method when changing state. Do not obtain the config and change the state directly, doing so will make the plugin config inconsistent with the internal state.
    
    Specified by:
    
    setEnabled in interface StoragePluginRegistry
    
    Parameters:
    
    name - name of the plugin
    
    enable - true to enable the plugin, false to disable
    
    Throws:
    
    StoragePluginRegistry.PluginNotFoundException - if the plugin is not found
    
    StoragePluginRegistry.PluginException - if the plugin name is not valid or if enabling a plugin and the plugin is not valid
  - getStoredConfig
```
public StoragePluginConfig getStoredConfig(String name)
```
    Configs are obtained from the persistent store. This method is called only by the UI to edit a stored plugin; so no benefit to using the cache. We also want a plugin even if it is disabled, and disabled plugins do not reside in the cache.
    Note that each call (depending on the store implementation) may return a distinct instance of the config. The instance will be equal (unless the stored version changes.) However, other versions of the store may return the same instance as is in the cache. So, do not modify the returned config. To modify the config, call copyConfig(String) instead.
    
    Specified by:
    
    getStoredConfig in interface StoragePluginRegistry
  - copyConfig
```
public StoragePluginConfig copyConfig(String name)
                               throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Copy a stored config so that it can be modified.
    Never modify a config stored in the registry! Configs are keyed by name and value; getting a config, then modifying it, will cause the value maps to become out of sync.
    
    Specified by:
    
    copyConfig in interface StoragePluginRegistry
    
    Parameters:
    
    name - name of the storage plugin config to copy
    
    Returns:
    
    a copy of the config
    
    Throws:
    
    StoragePluginRegistry.PluginException - if the name is undefined
  - encode
```
public String encode(StoragePluginConfig config)
```
    Specified by:
    
    encode in interface StoragePluginRegistry
  - encode
```
public String encode(String name)
              throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Return a config encoded as JSON.
    
    Specified by:
    
    encode in interface StoragePluginRegistry
    
    Throws:
    
    StoragePluginRegistry.PluginException - if the plugin is undefined
  - decode
```
public StoragePluginConfig decode(String json)
                           throws StoragePluginRegistry.PluginEncodingException
```
    Description copied from interface: StoragePluginRegistry
    
    Return a config decoded from JSON.
    
    Specified by:
    
    decode in interface StoragePluginRegistry
    
    Throws:
    
    StoragePluginRegistry.PluginEncodingException - if the JSON is invalid
  - putJson
```
public void putJson(String name,
                    String json)
             throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Put a storage plugin config from JSON. Validates the JSON and the resulting storage plugin. Use this form for JSON received from the UI or other external source.
    
    Specified by:
    
    putJson in interface StoragePluginRegistry
    
    Throws:
    
    StoragePluginRegistry.PluginException - if the underlying StoragePluginRegistry.validatedPut(String, StoragePluginConfig) fails
  - copyConfig
```
public StoragePluginConfig copyConfig(StoragePluginConfig orig)
```
    Description copied from interface: StoragePluginRegistry
    
    Copy the given storage plugin config so it may be modified.
    
    Specified by:
    
    copyConfig in interface StoragePluginRegistry
    
    Parameters:
    
    orig - the storage plugin config to copy
    
    Returns:
    
    the copy
  - getDefinedConfig
```
public StoragePluginConfig getDefinedConfig(String name)
```
    Description copied from interface: StoragePluginRegistry
    
    Retrieve an available configuration. Returns enabled stored plugins and system plugins. These configs are those that can be used to plan a query.
    
    Specified by:
    
    getDefinedConfig in interface StoragePluginRegistry
  - getPlugin
```
public StoragePlugin getPlugin(String name)
                        throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Get a plugin by name. Create it based on the PStore saved definition if it doesn't exist.
    
    Specified by:
    
    getPlugin in interface StoragePluginRegistry
    
    Parameters:
    
    name - The name of the plugin
    
    Returns:
    
    The StoragePlugin instance.
    
    Throws:
    
    StoragePluginRegistry.PluginException - if plugin cannot be obtained
  - getPlugin
```
public StoragePlugin getPlugin(StoragePluginConfig config)
                        throws ExecutionSetupException
```
    Specified by:
    
    getPlugin in interface StoragePluginRegistry
    
    Throws:
    
    ExecutionSetupException
  - getPluginByConfig
```
public StoragePlugin getPluginByConfig(StoragePluginConfig config)
                                throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Get a plugin by configuration. If it doesn't exist, create it.
    
    Specified by:
    
    getPluginByConfig in interface StoragePluginRegistry
    
    Parameters:
    
    config - The configuration for the plugin.
    
    Returns:
    
    The StoragePlugin instance.
    
    Throws:
    
    StoragePluginRegistry.PluginException - if plugin cannot be obtained
  - remove
```
public void remove(String name)
            throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Remove a plugin by name
    
    Specified by:
    
    remove in interface StoragePluginRegistry
    
    Parameters:
    
    name - The name of the storage plugin to remove
    
    Throws:
    
    StoragePluginRegistry.PluginException
  - storedConfigs
```
public Map<String,StoragePluginConfig> storedConfigs()
```
    Description copied from interface: StoragePluginRegistry
    
    Returns a set of all stored plugin configurations, directly from the persistent store. Note: the actual configs may reside in the cache; make a copy before making any changes.
    
    Specified by:
    
    storedConfigs in interface StoragePluginRegistry
    
    Returns:
    
    map of stored plugin configurations
  - storedConfigs
```
public Map<String,StoragePluginConfig> storedConfigs(StoragePluginRegistry.PluginFilter filter)
```
    Description copied from interface: StoragePluginRegistry
    
    Return a possibly-filtered set of plugins from the persistent store.
    
    Specified by:
    
    storedConfigs in interface StoragePluginRegistry
  - enabledConfigs
```
public Map<String,StoragePluginConfig> enabledConfigs()
```
    Description copied from interface: StoragePluginRegistry
    
    Returns a copy of the set of enabled stored plugin configurations. The registry is refreshed against the persistent store prior to building the map.
    
    Specified by:
    
    enabledConfigs in interface StoragePluginRegistry
    
    Returns:
    
    map of enabled, stored plugin configurations
  - putFormatPlugin
```
public void putFormatPlugin(String pluginName,
                            String formatName,
                            FormatPluginConfig formatConfig)
                     throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Safe way to add or remove a format plugin config from a stored file system configuration. Makes a copy of the config, adds/removes the format plugin, and updates the persistent store with the copy.
    
    Specified by:
    
    putFormatPlugin in interface StoragePluginRegistry
    
    Parameters:
    
    pluginName - name of the file system storage plugin config to modify
    
    formatName - name of the format plugin to modify
    
    formatConfig - if null, removes the plugin, if non-null updates the format plugin config with this value
    
    Throws:
    
    StoragePluginRegistry.PluginException - if the storage plugin is undefined or is not a file format plugin
  - getFormatPluginByConfig
```
public FormatPlugin getFormatPluginByConfig(StoragePluginConfig storageConfig,
                                            FormatPluginConfig formatConfig)
                                     throws StoragePluginRegistry.PluginException
```
    Description copied from interface: StoragePluginRegistry
    
    Get the Format plugin for the FileSystemPlugin associated with the provided storage config and format config.
    
    Specified by:
    
    getFormatPluginByConfig in interface StoragePluginRegistry
    
    Parameters:
    
    storageConfig - The storage config for the associated FileSystemPlugin
    
    formatConfig - The format config for the associated FormatPlugin
    
    Returns:
    
    A FormatPlugin instance
    
    Throws:
    
    StoragePluginRegistry.PluginException - if plugin cannot be obtained
  - getFormatPlugin
```
public FormatPlugin getFormatPlugin(StoragePluginConfig storageConfig,
                                    FormatPluginConfig formatConfig)
                             throws ExecutionSetupException
```
    Specified by:
    
    getFormatPlugin in interface StoragePluginRegistry
    
    Throws:
    
    ExecutionSetupException
  - getSchemaFactory
```
public SchemaFactory getSchemaFactory()
```
    Description copied from interface: StoragePluginRegistry
    
    Get the Schema factory associated with this storage plugin registry.
    
    Specified by:
    
    getSchemaFactory in interface StoragePluginRegistry
    
    Returns:
    
    A SchemaFactory that can register the schemas associated with this plugin registry.
  - iterator
```
public Iterator<Map.Entry<String,StoragePlugin>> iterator()
```
    Specified by:
    
    iterator in interface Iterable<Map.Entry<String,StoragePlugin>>
  - close
```
public void close()
           throws Exception
```
    Specified by:
    
    close in interface AutoCloseable
    
    Throws:
    
    Exception
  - mapper
```
public com.fasterxml.jackson.databind.ObjectMapper mapper()
```
    Description copied from interface: StoragePluginRegistry
    
    Object mapper to read/write the JSON form of a plugin. config.
    
    Specified by:
    
    mapper in interface StoragePluginRegistry
  - resolve
```
public <T extends StoragePlugin> T resolve(StoragePluginConfig storageConfig,
                                           Class<T> desired)
```
    Specified by:
    
    resolve in interface StoragePluginRegistry
  - resolveFormat
```
public <T extends FormatPlugin> T resolveFormat(StoragePluginConfig storageConfig,
                                                FormatPluginConfig formatConfig,
                                                Class<T> desired)
```
    Description copied from interface: StoragePluginRegistry
    
    Resolve a storage plugin given a storage plugin config. Call from within a file storage plugin to resolve the plugin.
    
    Specified by:
    
    resolveFormat in interface StoragePluginRegistry
    
    Type Parameters:
    
    T - the required type of the plugin
    
    Parameters:
    
    storageConfig - storage plugin config
    
    formatConfig - format plugin config
    
    desired - desired target class
    
    Returns:
    
    the storage plugin
  - availablePlugins
```
public Set<String> availablePlugins()
```
    Description copied from interface: StoragePluginRegistry
    
    Returns the set of available plugin names. Includes system plugins and enabled stored plugins.
    
    Specified by:
    
    availablePlugins in interface StoragePluginRegistry

Class StoragePluginRegistryImpl

Concurrency

Ephemeral Plugins

Caveats

Error Handling

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.drill.exec.store.StoragePluginRegistry

Field Summary

Fields inherited from interface org.apache.drill.exec.store.StoragePluginRegistry

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface java.lang.Iterable

Constructor Detail

StoragePluginRegistryImpl

Method Detail

init

copyPluginStatus

put

validatedPut

setEnabled

getStoredConfig

copyConfig

encode

encode

decode

putJson

copyConfig

getDefinedConfig

getPlugin

getPlugin

getPluginByConfig

remove

storedConfigs

storedConfigs

enabledConfigs

putFormatPlugin

getFormatPluginByConfig

getFormatPlugin

getSchemaFactory

iterator

close

mapper

resolve

resolveFormat

availablePlugins