Apache Drill 0.9.0 Release Notes

It has been about a month since the release of Drill 0.8, which included more than 240 improvements. Today we’re happy to announce the availability of Drill 0.9, providing additional enhancements and bug fixes. In fact, this release includes 200 resolved JIRAs. Some of the noteworthy features in Drill 0.9 are:

• Authentication (DRILL-2674). Drill now supports username/password authentication through the Java and C++ clients, as well as JDBC and ODBC. On the server-side, Drill leverages Linux PAM to securely validate the credentials. Users can choose to use an external user directory such as Active Directory or LDAP. To enable authentication, set the security.user.auth option in drill-override.conf.
• Impersonation (DRILL-2363). Queries now execute and access resources using the identity of the user who submitted the query. Previously, all queries would run as the same user (eg, drill). With the new impersonation capability, the query will fail if the submitting user does not have permission to read the requested file(s) in the distributed file system. To enable impersonation, set the drill.exec.impersonation option in drill-override.conf.
• Ownership chaining. Drill now allows views with different owners to be chained. This represents a very flexible access control solution. For example, an administrator with access to raw, sensitive data could create a view called masked which would expose only a subset of the data to other users. The administrator would enable users to read the masked view but not the raw data. Note that Drill provides an option max_chained_user_hops that specifies how many ownership changed are allowed in a chain, thereby providing administrators (or data stewards) more control over sharing of data.
• MongoDB authentication (DRILL-1502). Drill can now connect to a MongoDB cluster that requires authentication.
• Extended JSON datatypes. Our friends at MongoDB invented extended JSON - a set of extensions to the JSON format for supporting additional data types. We decided to embrace extended JSON in Drill. For example, standard JSON doesn’t have a time type, so a time could be represented as either a string or a number: {"foo": "19:20:30.450Z"} is just a string. With extended JSON, the $time qualifier can be used to specify that foo is a time {"foo": {"$time": "19:20:30.450Z"}}. We now support a number of qualifiers including $bin, $date, $time, $interval, $numberLong and $dateDay (see the example). We’re in the process of adding some additional qualifiers to make sure that all of MongoDB’s extended types are supported (this is particularly important when querying data in MongoDB).
• Avro support (DRILL-1512). Drill can now read Avro files. This patch was contributed by Andrew Selden at Elastic.co (formerly known as Elasticsearch).
• Improved error messages (DRILL-2675 and more). It can be challenging for a complex distributed system like Drill to translate low-level internal conditions into actionable messages to the user. This release includes several enhancements that enable Drill to accomplish just that in a variety of cases.
• Parquet and Calcite enhancements (DRILL-1410 and DRILL-1384). Drill isn’t a traditional query engine - it’s the first analytical query engine with a JSON data model. This has required us to enhance Parquet (our columnar format) and Calcite (our SQL parser). These enhancements have now been contributed back to those projects, and Drill is using the latest versions which include these enhancements.

• New sys tables for memory and thread information (DRILL-2275). Drill includes two new sys tables that provide real-time metrics about memory utilization and threads on each of the nodes in the cluster. You can run a simple SELECT * to see what information is available:

    SELECT * FROM sys.drillmemory;
    SELECT * FROM sys.drillbitthreads;
  

• Support for very wide tables (DRILL-2739). Drill previously had some issues with tables that had more than 4095 colums. This limitation has been addressed.

You can now download Drill 0.9. As always, you can check out the official release notes for more details.

We’re gearing up for Drill’s 1.0 release later this month. Stay tuned!