Package org.apache.drill.exec.store.http.util

Class SimpleHttp

java.lang.Object
org.apache.drill.exec.store.http.util.SimpleHttp
All Implemented Interfaces:
AutoCloseable
public class SimpleHttp extends Object implements AutoCloseable
Performs the actual HTTP requests for the HTTP Storage Plugin. The core method is the getInputStream() method which accepts a url and opens an InputStream with that URL's contents.

  • Field Details

    • JSON_MEDIA_TYPE

      public static final okhttp3.MediaType JSON_MEDIA_TYPE

    • XML_MEDIA_TYPE

      public static final okhttp3.MediaType XML_MEDIA_TYPE

  • Constructor Details

    • SimpleHttp

      public SimpleHttp(HttpSubScan scanDefn, okhttp3.HttpUrl url, File tempDir, HttpProxyConfig proxyConfig, CustomErrorContext errorContext, Paginator paginator)

    • SimpleHttp

      public SimpleHttp(okhttp3.HttpUrl url, File tempDir, HttpProxyConfig proxyConfig, CustomErrorContext errorContext, Paginator paginator, PersistentTokenTable tokenTable, HttpStoragePluginConfig pluginConfig, HttpApiConfig endpointConfig, String connection, Map<String,String> filters, int rateLimit)
      This constructor does not have an HttpSubScan and can be used outside the context of the HttpStoragePlugin.
      Parameters:
      url - The URL for an HTTP request
      tempDir - Temp directory for caching
      proxyConfig - Proxy configuration for making API calls
      errorContext - The error context for error messages
      paginator - The Paginator object for pagination.
      tokenTable - The OAuth token table
      pluginConfig - HttpStoragePlugin configuration. The plugin obtains OAuth and timeout info from this config.
      endpointConfig - The
      connection - The name of the connection
      filters - A Key/value set of filters and values

  • Method Details

    • builder

      public static SimpleHttp.SimpleHttpBuilder builder()

    • setupHttpClient

      protected okhttp3.OkHttpClient setupHttpClient()
      Configures the OkHTTP3 server object with configuration info from the user.
      Returns:
      OkHttpClient configured server

    • url

      public String url()

    • addProxyInfo

      public static void addProxyInfo(okhttp3.OkHttpClient.Builder builder, HttpProxyConfig proxyConfig)
      Applies the proxy configuration to the OkHttp3 builder. This ensures that proxy configurations will be consistent across HTTP REST connections.
      Parameters:
      builder - The input OkHttp3 builder
      proxyConfig - The proxy configuration

    • getInputStream

      public InputStream getInputStream()
      Returns an InputStream based on the URL and config in the scanSpec. If anything goes wrong the method throws a UserException.
      Returns:
      An Inputstream of the data from the URL call. The caller is responsible for calling close() on the InputStream.

    • getResultsFromApiCall

      public String getResultsFromApiCall()

    • getProxySettings

      public static HttpProxyConfig getProxySettings(HttpStoragePluginConfig config, com.typesafe.config.Config drillConfig, okhttp3.HttpUrl url)

    • getResponseCode

      public int getResponseCode()
      Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.
      Returns:
      int value of the HTTP response code

    • getResponseMessage

      public String getResponseMessage()
      Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.
      Returns:
      int of HTTP response code

    • getResponseProtocol

      public String getResponseProtocol()
      Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.
      Returns:
      The HTTP response protocol

    • getResponseURL

      public String getResponseURL()
      Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.
      Returns:
      The HTTP response URL

    • buildPostBody

      public okhttp3.FormBody.Builder buildPostBody(Map<String,String> filters, String postBody)
      This function is used to push filters down to the post body rather than the URL query string. It will also add the static parameters to the post body as well.
      Parameters:
      filters - A HashMap of the filters and values
      postBody - The post body of static parameters.
      Returns:
      The post body builder with the filters and static parameters

    • decodedURL

      public static String decodedURL(okhttp3.HttpUrl url)
      Returns the URL-decoded URL. If the URL is invalid, return the original URL.
      Returns:
      Returns the URL-decoded URL

    • hasURLParameters

      public static boolean hasURLParameters(okhttp3.HttpUrl url)
      Returns true if the url has url parameters, as indicated by the presence of {param} in a url.
      Returns:
      True if there are URL params, false if not

    • getURLParameters

      public static List<String> getURLParameters(okhttp3.HttpUrl url)
      APIs are sometimes structured with parameters in the URL itself. For instance, to request a list of an organization's repositories in github, the URL is: https://api.github.com/orgs/{org}/repos, where you can replace the org with the actual organization name.
      Returns:
      A list of URL parameters enclosed by curly braces.

    • getDefaultParameterValue

      public static String getDefaultParameterValue(okhttp3.HttpUrl url, String parameter)
      This function is used to extract the default parameter supplied in a URL. For instance, if the supplied URL is http://someapi.com/path/{p1=foo}, the function will return foo. If there is not a matching parameter or no default value, the function will return null.
      Parameters:
      url - The URL containing a default parameter
      parameter - The parameter for which you need the value
      Returns:
      The value for the supplied parameter

    • mapURLParameters

      public static String mapURLParameters(okhttp3.HttpUrl url, Map<String,String> filters)
      Used for APIs which have parameters in the URL. This function maps the filters pushed down from the query into the URL. For example the API: github.com/orgs/{org}/repos requires a user to specify an organization and replace {org} with an actual organization. The filter is passed down from the query. Note that if a URL contains URL parameters and one is not provided in the filters, Drill will throw a UserException.
      Parameters:
      url - The HttpUrl containing URL Parameters
      filters - A CaseInsensitiveMap of filters
      Returns:
      A string of the URL with the URL parameters replaced by filter values

    • mapPositionalParameters

      public static String mapPositionalParameters(String rawUrl, List<String> params)

    • validateUrl

      public static boolean validateUrl(String url)
      Validates a URL.
      Parameters:
      url - The input URL. Should be a string.
      Returns:
      True of the URL is valid, false if not.

    • buildParameterList

      public static List<String> buildParameterList(NullableVarCharHolder[] inputReaders)
      Accepts a list of input readers and converts that into an ArrayList of Strings
      Parameters:
      inputReaders - The array of FieldReaders
      Returns:
      A List of Strings containing the values from the FieldReaders or null to indicate that at least one null is present in the arguments.

    • getEndpointConfig

      public static HttpApiConfig getEndpointConfig(String endpoint, HttpStoragePluginConfig pluginConfig)
      This function is used to obtain the configuration information for a given API in the HTTP UDF. If aliasing is enabled, this function will resolve aliases for connections.
      Parameters:
      endpoint - The name of the endpoint. Should be a String
      pluginConfig - The HttpStoragePluginConfig the configuration from the plugin
      Returns:
      The HttpApiConfig corresponding with the endpoint.

    • getStoragePlugin

      public static HttpStoragePlugin getStoragePlugin(String pluginName, DrillbitContext context)
      This function will return a HttpStoragePlugin for use in the HTTP UDFs. If user or public aliases are used, the function will resolve those aliases.
      Parameters:
      context - A DrillbitContext from the current query
      pluginName - A String of the plugin name. Note that the function will resolve aliases.
      Returns:
      A HttpStoragePlugin of the plugin.

    • apiCall

      public static SimpleHttp apiCall(HttpStoragePlugin plugin, HttpApiConfig endpointConfig, DrillbitContext context, List<String> args)
      This function makes an API call and returns a string of the parsed results. It is used in the http_get() UDF and retrieves all the configuration parameters contained in the storage plugin and endpoint configuration. The exception is pagination. This does not support pagination.
      Parameters:
      plugin - The HTTP storage plugin upon which the API call is based.
      endpointConfig - The configuration of the API endpoint upon which the API call is based.
      context - DrillbitContext The context from the current query
      args - An optional list of parameter arguments which will be included in the URL
      Returns:
      A String of the results.

    • getRequestAndStringResponse

      public static String getRequestAndStringResponse(String url)

    • getRequestAndStreamResponse

      public static InputStream getRequestAndStreamResponse(String url)
      Parameters:
      url -
      Returns:
      an input stream which the caller is responsible for closing.

    • makeSimpleGetRequest

      public static okhttp3.ResponseBody makeSimpleGetRequest(String url) throws IOException
      Parameters:
      url -
      Returns:
      response body which the caller is responsible for closing.
      Throws:
      IOException

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable