Apache Drill 1.18.0 Release Notes

Release date: September 05, 2020

Today, we’re happy to announce the availability of Drill 1.18.0. You can download it here.

New Feature

  • [DRILL-6552] - Drill Metadata management “Drill Metastore”
  • [DRILL-7233] - Format Plugin for HDF5
  • [DRILL-7359] - Add support for DICT type in RowSet Framework
  • [DRILL-7437] - Storage Plugin for Generic HTTP REST API
  • [DRILL-7607] - Dynamic credit based flow control
  • [DRILL-7656] - Support injecting BufferManager into UDF
  • [DRILL-7706] - Drill RDBMS Metastore
  • [DRILL-5956] - Add Storage Plugin for Apache Druid

Improvement

  • [DRILL-6599] - Setup continuous integration for Drill builds on Docker Hub
  • [DRILL-6600] - Create Docker Official Repository for Drill
  • [DRILL-6604] - Upgrade Drill Hive client to Hive3.1 version
  • [DRILL-6738] - validationQuery option for JDBC storage plugin
  • [DRILL-6832] - Remove old “unmanaged” sort implementation
  • [DRILL-7361] - Add Map (Dict) support for schema provisioning using file
  • [DRILL-7466] - Configurable number of connection in JDBC storage connection pool
  • [DRILL-7467] - Jdbc plugin enhancements and fixes
  • [DRILL-7478] - Add Pull Request Template
  • [DRILL-7503] - Refactor project operator
  • [DRILL-7506] - Simplify code gen error handling
  • [DRILL-7507] - Convert fragment interrupts to exceptions
  • [DRILL-7514] - Update Apache POI to Latest Version
  • [DRILL-7547] - More secure storage for mongodb credentials
  • [DRILL-7562] - Support HTTP Basic authentication
  • [DRILL-7572] - JSON structure parser
  • [DRILL-7573] - Support text file for user authentication instead of using PAM
  • [DRILL-7574] - Generalize projection parser
  • [DRILL-7575] - FormSecurityHandler class name is spelled incorrectly
  • [DRILL-7576] - Fail fast in operators
  • [DRILL-7583] - Remove STOP status in favor of fail-fast
  • [DRILL-7590] - Refactor plugin registry
  • [DRILL-7599] - Move Function to Access Drill Temp Directory to Utility Library
  • [DRILL-7600] - Move Compress File Function to Test Utility
  • [DRILL-7601] - Shift column conversion to reader from scan framework
  • [DRILL-7603] - Allow setting default schema using REST API / Web UI
  • [DRILL-7605] - Web UI should remember last Query text & settings
  • [DRILL-7610] - Allow user to specify table schema in Metastore
  • [DRILL-7618] - Extend configuration options for embedded Jetty server
  • [DRILL-7631] - Updates to the Json Structure Parser
  • [DRILL-7632] - Improve user exception formatting
  • [DRILL-7633] - Fixes for union and repeated list accessors
  • [DRILL-7634] - Rollup of code cleanup changes
  • [DRILL-7637] - Add an option to retrieve MapR SSL truststore/keystore credentials using MapR Web Security Manager
  • [DRILL-7640] - EVF-based JSON Loader
  • [DRILL-7641] - Convert Excel Reader to Use Streaming Reader
  • [DRILL-7650] - Add option to enable Jetty’s dump for troubleshooting
  • [DRILL-7652] - Add time_bucket() Function for Time Series Analysis
  • [DRILL-7663] - Code refactor to DefaultFunctionResolver
  • [DRILL-7665] - Add UNION to schema parser
  • [DRILL-7668] - Allow Time Bucket Function to Accept Floats and Timestamps
  • [DRILL-7672] - Make metadata type required when reading from / writing into Drill Metastore
  • [DRILL-7678] - Update Yauaa dependency
  • [DRILL-7680] - Move UDF projects before plugins in contrib
  • [DRILL-7683] - Add “message parsing” to new JSON loader
  • [DRILL-7692] - Select default schema from enabled storage plugins in query page
  • [DRILL-7693] - Upgrade Protobuf to 3.11.1
  • [DRILL-7696] - EVF v2 Scan Schema Resolution
  • [DRILL-7701] - EVF V2 Scan Framework
  • [DRILL-7703] - Support for 3+D arrays in EVF JSON loader
  • [DRILL-7705] - Update jQuery and Bootstrap libraries
  • [DRILL-7711] - Add data path, parameter filter pushdown to HTTP plugin
  • [DRILL-7716] - Create Format Plugin for SPSS Files
  • [DRILL-7717] - Support Mongo extended types in V2 JSON loader
  • [DRILL-7723] - Add Excel Metadata as Implicit Fields
  • [DRILL-7724] - Refactor metadata controller batch
  • [DRILL-7725] - Updates to EVF2
  • [DRILL-7730] - Reduce overhead of web queries displayed in HTML
  • [DRILL-7734] - Revise the result set reader
  • [DRILL-7739] - Allow implicit casts from required to nullable data type
  • [DRILL-7746] - Add REST API Ability to Accept CSV Input

Sub-task

  • [DRILL-7229] - Add scripts to the release folder in Drill Repo
  • [DRILL-7230] - Add README.md with instructions for release
  • [DRILL-7274] - Introduce ANALYZE TABLE statements
  • [DRILL-7330] - Implement metadata usage for text format plugin
  • [DRILL-7386] - Fix count and typeOf functions for struct type
  • [DRILL-7454] - Convert the Avro format plugin to use EVF
  • [DRILL-7477] - Allow passing table function parameters into ANALYZE statement
  • [DRILL-7625] - Add options for SslContextFactory
  • [DRILL-7626] - Add ability to set HTTP response headers
  • [DRILL-7644] - Log SSL protocol version at Drill start up
  • [DRILL-7782] - Update the copyright year in NOTICE

Bug

  • [DRILL-4527] - Remove unnecessary code: DrillAvgVarianceConvertlet.java
  • [DRILL-5733] - Unable to SELECT from parquet file with Hadoop 2.7.4
  • [DRILL-6168] - Table functions do not “inherit” default configuration
  • [DRILL-6215] - Use prepared statement instead of Statement in JdbcRecordReader class
  • [DRILL-6905] - ClassCastException happens when combining filters with numeric and varchar literals
  • [DRILL-7203] - Back button for failed query does not return on Query page
  • [DRILL-7311] - Partial fixes for empty batch bugs
  • [DRILL-7340] - Filter is not pushed to JDBC database when several databases are used in the query
  • [DRILL-7415] - Information schema query fails for postgres foreign tables
  • [DRILL-7429] - Wrong column order when selecting complex data using Hive storage plugin.
  • [DRILL-7459] - Fetch size does not work on Postgres JDBC plugin
  • [DRILL-7461] - Do not pass ClassNotFoundException into SQLNonTransientConnectionException cause when checking that Drill is run in embedded mode
  • [DRILL-7491] - Incorrect count() returned for complex types in parquet
  • [DRILL-7495] - Excel Reader Not Parsing Dates Correctly in First Column
  • [DRILL-7497] - Fix warnings when starting Drill on Windows using Java 11
  • [DRILL-7502] - Incorrect/invalid codegen for typeof() with UNION
  • [DRILL-7505] - PCAP Plugin Fails on IPv6 Packets
  • [DRILL-7509] - Incorrect TupleSchema is created for DICT column when querying Parquet files
  • [DRILL-7518] - Parquet INT64 Nullable Type Support
  • [DRILL-7527] - DROP METADATA doesn’t work with table name starting with ‘/’ inside workspace
  • [DRILL-7529] - Building depends on poorly configured uncommon maven repositories
  • [DRILL-7549] - Fix validation error when querying absent sub folder in embedded mode
  • [DRILL-7565] - ANALYZE TABLE … REFRESH METADATA does not work for empty Parquet files
  • [DRILL-7578] - HDF5 Metadata Queries Fail with Large Files
  • [DRILL-7582] - Drill docker Web UI doesn’t show resources usage information if map the container to a non-default port
  • [DRILL-7586] - drill-hive-exec-shaded contains commons-lang3 version 3.1
  • [DRILL-7587] - Query with VALUES clause returns the wrong result when planner.slice_target is small
  • [DRILL-7589] - TestDynamicUDFSupport fails on GitHub Actions
  • [DRILL-7596] - drill-format-esri module uses 1.16.0 version of org.apache.drill.contrib:drill-udfs
  • [DRILL-7615] - UNION ALL query returns the wrong result for the decimal value
  • [DRILL-7617] - Disabled storage plugins configuration is not displayed on Web UI
  • [DRILL-7619] - Metrics is not displayed due to incorrect endpoint link on the Drill index page
  • [DRILL-7620] - Storage plugin update page shows that a plugin is disabled though it is actually enabled.
  • [DRILL-7622] - Compilation error when using HLL / TDigest with group by clause
  • [DRILL-7623] - Link error is displayed at the log content page on Web UI
  • [DRILL-7624] - When Hive plugin is enabled with default config, cannot execute any SQL query
  • [DRILL-7628] - Fix Mongo tests broken after the fix for DRILL-7547
  • [DRILL-7629] - Parquet MAP field support missing in recent stable release (?)
  • [DRILL-7630] - Add additional types into SchemaParser
  • [DRILL-7635] - Mongo tests fail for profiles with non-file default file system
  • [DRILL-7643] - Fix issues with using columns with the same name as reserved keyword
  • [DRILL-7648] - Scrypt j_security_check works without security headers
  • [DRILL-7649] - Replace maprfs.version property usage by mapr.release.version
  • [DRILL-7673] - View set query fails with NPE for non-existing option
  • [DRILL-7675] - Very slow performance and Memory exhaustion while querying on very small dataset of parquet files
  • [DRILL-7694] - Register drill.queries.* counter metrics on Drillbit startup
  • [DRILL-7699] - Update javax.validation Dependency
  • [DRILL-7707] - Unable to analyze table metadata is it resides in non-writable workspace
  • [DRILL-7708] - Downgrade maven from 3.6.3 to 3.6.0
  • [DRILL-7710] - Fix TestMetastoreCommands#testDefaultSegment test
  • [DRILL-7726] - Boost requirement is incorrect
  • [DRILL-7738] - Fix TestDynamicUDFSupport failure for GitHub Actions
  • [DRILL-7741] - Columns are missing when using convert_from function
  • [DRILL-7749] - Drill-on-Yarn Application Master UI is broken due to bootstrap update
  • [DRILL-7750] - Drill fails to read KeyStore password from Credential provider
  • [DRILL-7759] - Code compilation exception for queries containing (untyped) NULL
  • [DRILL-7761] - Drill fails with OOM for the case of large filter conditions
  • [DRILL-7773] - Incorrect conversion result to TIME_EPOCH_BE
  • [DRILL-7774] - IS NOT NULL predicate fails with NPE for the case of non-existing columns with non-deterministic expression
  • [DRILL-7786] - Apache drill fails to execute when excel file has datetime column

Task

  • [DRILL-6580] - Upgrade Curator libraries
  • [DRILL-7220] - Create a release package in Drill repo with automated scripts and instructions
  • [DRILL-7406] - Update Calcite to 1.21.0
  • [DRILL-7496] - Update Dockerfile to publish release images under Apache Docker Hub
  • [DRILL-7504] - Upgrade Parquet library to 1.11.0
  • [DRILL-7528] - Update Avro format plugin documentation
  • [DRILL-7530] - Fix class names in loggers
  • [DRILL-7540] - Update Calcite.md doc
  • [DRILL-7543] - Use GitHub Actions for CI
  • [DRILL-7544] - Upgrade Iceberg version to support Parquet 1.11.0
  • [DRILL-7548] - Updating Environment.md to Skip Tests
  • [DRILL-7561] - Document authentication for REST API
  • [DRILL-7570] - Fix unstable statistics tests
  • [DRILL-7580] - Broken links in apache drill documentation to https://drill.apache.org/docs/starting-the-web-console/
  • [DRILL-7584] - Misspelled method name in MongoStoragePluginConfig (getMongoCrendials)
  • [DRILL-7592] - Add missing licenses and update plugins exclusion list
  • [DRILL-7594] - Remove unused DrillStoreRel
  • [DRILL-7614] - Add helpful tips to Testing.md
  • [DRILL-7627] - Update MySql version for JdbcStoragePlugin tests and cache ~/.embedmysql
  • [DRILL-7639] - Replace DBCP2 with HikariCP in RDBMS (JDBC) plugin
  • [DRILL-7651] - Increase timeout for TestLargeFileCompilation to avoid GitHub Action failures
  • [DRILL-7654] - Add support for JDK 14
  • [DRILL-7702] - Update zookeeper, curator, guava, jetty-server, libthrift, httpclient, commons-compress and httpdlog-parser
  • [DRILL-7704] - Update Maven dependency to 3.6.3
  • [DRILL-7713] - Upgrade misc libraries which outdated versions have reported vulnerabilities
  • [DRILL-7744] - Move Filters from HTTP Storage Plugin to Drill Core
← Apache Drill 1.19.0 Release NotesApache Drill 1.17.0 Release Notes →