Apache Drill 1.15.0 Release Notes

Release date: December 31, 2018

Today, we’re happy to announce the availability of Drill 1.15.0. You can download it here.

New Features and Improvements

This release of Drill provides the following new features and improvements:

The following sections provide a complete list of all the fixes and improvements in Drill 1.15:

Sub-task

  • [DRILL-3988] - Create a sys.functions table to expose available Drill functions
  • [DRILL-6461] - Add Basic Data Correctness Unit Tests
  • [DRILL-6617] - Planner Side changed to propagate $drill_implicit_field$ information
  • [DRILL-6618] - Unnest changes for implicit column
  • [DRILL-6619] - Lateral changes for implicit column
  • [DRILL-6660] - Exchange operators Analysis
  • [DRILL-6719] - Moving Common Spilling Code Into a Seperate Utility
  • [DRILL-6777] - Setup CircleCI configs for Drill
  • [DRILL-6798] - Planner changes to support semi-join
  • [DRILL-6876] - Enable CircleCI builds for JDK 9-11

Bug

  • [DRILL-1291] - If typing up botton in SQLline to show previous SQL in history, it only shows last line.
  • [DRILL-2108] - Running SQLLine with bad command syntax yields index exception (not usage message)
  • [DRILL-2299] - Case sensitivity: Storage plugin names and workspace names are case sensitive
  • [DRILL-2597] - Sqlline fails when script contains comments
  • [DRILL-3926] - TPCH Concurrency Scale tests hit ChannelClosedException
  • [DRILL-3933] - Error execute select command line sqlline -u -q
  • [DRILL-3950] - CAST(...) * (Interval Constant) gives Internal Exception
  • [DRILL-4546] - mvn deploy pushes the same zip artifact twice
  • [DRILL-4897] - NumberFormatException in Drill SQL while casting to BIGINT when its actually a number
  • [DRILL-4984] - Limit 0 raises NullPointerException on JDBC storage sources
  • [DRILL-5141] - Difference in output date format on sqlline vs Drill's Web UI
  • [DRILL-5365] - Enforce Immutability of DrillFileSystem
  • [DRILL-5571] - Unable to cancel running queries from Web UI
  • [DRILL-5578] - Drill fails on date functions in 'where clause' when queried on a JDBC Storage plugin
  • [DRILL-5782] - Web UI: do not attempt to build visualized plan when plan is absent
  • [DRILL-6039] - drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit
  • [DRILL-6069] - Hash agg operator requires large memory amount when planner.width.max_per_node is large
  • [DRILL-6211] - Optimizations for SelectionVectorRemover
  • [DRILL-6246] - Build Failing in jdbc-all artifact
  • [DRILL-6349] - Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
  • [DRILL-6409] - Query Failed: An Error Occurred
  • [DRILL-6410] - Memory leak in Parquet Reader during cancellation
  • [DRILL-6453] - TPC-DS query 72 has regressed
  • [DRILL-6517] - IllegalStateException: Record count not set for this vector container
  • [DRILL-6563] - TPCDS query 10 has regressed
  • [DRILL-6566] - Jenkins Regression: TPCDS query 66 fails with RESOURCE ERROR: One or more nodes ran out of memory while executing the query. AGGR OOM at First Phase.
  • [DRILL-6567] - Jenkins Regression: TPCDS query 93 fails with INTERNAL_ERROR ERROR: java.lang.reflect.UndeclaredThrowableException.
  • [DRILL-6569] - Jenkins Regression: TPCDS query 19 fails with INTERNAL_ERROR ERROR: Can not read value at 2 in block 0 in file maprfs:///drill/testdata/tpcds_sf100/parquet/store_sales/1_13_1.parquet
  • [DRILL-6589] - Push transitive closure generated predicates past aggregates
  • [DRILL-6625] - Intermittent failures in Kafka unit tests
  • [DRILL-6630] - Extra spaces are ignored while publishing results in Drill Web UI
  • [DRILL-6631] - Wrong result from LateralUnnest query with aggregation and order by
  • [DRILL-6638] - Fix TestE2EUnnestAndLateral tests introduced with DRILL-6546
  • [DRILL-6639] - Exception happens while displaying operator profiles for some queries
  • [DRILL-6641] - TestConvertCountToDirectScan.ensureCorrectCountWithMissingStatistics() test fails randomly
  • [DRILL-6649] - Query with unnest of column from nested subquery fails
  • [DRILL-6651] - Compilation error in IDE due to missing package name
  • [DRILL-6653] - Unsupported Schema change exception where there is no schema change in lateral Unnest queries
  • [DRILL-6654] - Data verification failure with lateral unnest query having filter in and order by
  • [DRILL-6657] - Unnest reports one batch less than the actual number of batches
  • [DRILL-6663] - Shutdown not working when IP is used to load WebUI in secure cluster
  • [DRILL-6666] - Doc link to AOL data set is broken
  • [DRILL-6670] - Error in parquet record reader - previously readable file fails to be read in 1.14
  • [DRILL-6671] - Multi level lateral unnest join is throwing an exception during materializing the plan.
  • [DRILL-6674] - Minor fixes to avoid auto boxing cost in logging in LateralJoinBatch
  • [DRILL-6685] - Error in parquet record reader
  • [DRILL-6688] - Data batches for Project operator exceed the maximum specified
  • [DRILL-6693] - When a query is started from Drill Web Console, the UI becomes inaccessible until the query is completed
  • [DRILL-6694] - NPE in UnnestRecordBatch when query uses a column name not present in data
  • [DRILL-6696] - IOBE in Operator Metric Registry
  • [DRILL-6703] - Query with complex expressions in lateral and unnest fails with CannotPlanException
  • [DRILL-6705] - Fix various failures in Crypto / Network / Phonetic functions when illegal input is given
  • [DRILL-6706] - Query with 10-way hash join fails with NullPointerException
  • [DRILL-6709] - Batch statistics logging utility needs to be extended to mid-stream operators
  • [DRILL-6710] - Disallow negative scale for VarDecimal
  • [DRILL-6712] - Creation of jdbc storage plugin fails with NoSuchMethod
  • [DRILL-6713] - Infinite loop when retrieving decimal value with scale = -1
  • [DRILL-6717] - lower and upper functions not works with national charactes
  • [DRILL-6721] - SchemalessScan cannot be deserialized by Jackson
  • [DRILL-6726] - Drill fails to query views created before DRILL-6492 when impersonation is enabled
  • [DRILL-6732] - Queries are runnable on disable plugins
  • [DRILL-6733] - Unit tests from KafkaFilterPushdownTest are failing in some environments.
  • [DRILL-6746] - Query can hang when PartitionSender task thread sees a connection failure while sending data batches to remote fragment
  • [DRILL-6747] - Empty Probe Side in Right or Inner Join Causes IOB.
  • [DRILL-6748] - Docs for configuring Drill to use S3 Instance Profiles
  • [DRILL-6749] - Fixing broken links to CTAS and Explain Commands
  • [DRILL-6753] - Fix show files command to return result the same way as before
  • [DRILL-6759] - CSV 'columns' array is incorrectly case sensitive
  • [DRILL-6760] - The Profiles page does not retain the original exception if the query fails
  • [DRILL-6762] - Dynamic UDFs registered on one Drillbit are not visible on other Drillbits
  • [DRILL-6764] - Query fails with IOB when Unnest has reference to deep nested field like (t.c_orders.o_lineitems)
  • [DRILL-6766] - Lateral Unnest query : IllegalStateException - rowId in right batch of lateral is smaller than rowId in left batch being processed
  • [DRILL-6769] - Sqlline doesn't see line endings when it prints first row fo the explain plan results table
  • [DRILL-6770] - Queries on MapR-DB JSON tables fail with UnsupportedOperationException: Getting number of rows for tablet not supported
  • [DRILL-6771] - Queries on Hive 2.3.x fails with SYSTEM ERROR: ArrayIndexOutOfBoundsException
  • [DRILL-6773] - The renamed schema with aliases is not shown for queries on empty directories
  • [DRILL-6775] - The schema for empty output is not shown in Drill Web UI
  • [DRILL-6776] - Drill Web UI takes long time for first time load in network isolated environment
  • [DRILL-6783] - CAST string literal as INTERVAL MONTH/YEAR works inconsistently when selecting from a table with multiple rows
  • [DRILL-6785] - DataClient is using RootAllocator in the bootstrap instead of dataPool
  • [DRILL-6787] - Update Spnego webpage
  • [DRILL-6788] - Intermittent unit test failure TestDrillbitResilience.failsWhenParsing: Query state should be FAILED (and not COMPLETED)
  • [DRILL-6793] - FragmentExecutor cannot send its final state for the case when RootExec root wasn't initialized
  • [DRILL-6797] - Split function index extraction fails when result has no rows
  • [DRILL-6809] - Handle repeated map in schema inference
  • [DRILL-6810] - Disable NULL_IF_NULL NullHandling for functions with ComplexWriter
  • [DRILL-6811] - Fix type inference to return correct data mode for boolean functions
  • [DRILL-6819] - Remove invisible "back" link on query results page in Drill WebUI
  • [DRILL-6824] - Drill Query on MapRDB JSON table failing on schema SchemaChangeException, the only distinct Values are NULL and Text
  • [DRILL-6828] - Hit UnrecognizedPropertyException when run tpch queries
  • [DRILL-6837] - Missing reference for dynamically created Javascript library
  • [DRILL-6838] - Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
  • [DRILL-6843] - SchemaBuilder class does not contain correct method names
  • [DRILL-6844] - Query with ORDER BY DESC on indexed column does not pick secondary index
  • [DRILL-6848] - Duration panel in a query profile's WebUI does not open
  • [DRILL-6850] - JDBC integration tests failures
  • [DRILL-6853] - Parquet Complex Reader for nested schema should have configurable memory or max records to fetch
  • [DRILL-6857] - Limit is not being pushed into scan when selecting from a parquet file with multiple row groups.
  • [DRILL-6858] - INFORMATION_SCHEMA.`FILES` table not show any files if storage contains "write only" folder
  • [DRILL-6861] - Hash-Join: Spilled partitions are skipped following an empty probe side
  • [DRILL-6863] - Drop table is not working if path within workspace starts with '/'
  • [DRILL-6865] - Query returns wrong result when filter pruning happens
  • [DRILL-6867] - WebUI Query editor cursor position
  • [DRILL-6874] - CTAS from json to parquet is not working on S3 storage
  • [DRILL-6877] - NPE when starting Drill
  • [DRILL-6878] - Index Plan Test failures after DRILL-6833
  • [DRILL-6882] - Handle the cases where RowKeyJoin's left pipeline being called multiple times.
  • [DRILL-6883] - Reset button in /options tab is not resetting the value
  • [DRILL-6886] - Cancel button does not show for running Query in WebUI
  • [DRILL-6887] - Fix FunctionInitializerTest.init() failure
  • [DRILL-6900] - maven-release-plugin failure on the release:perform stage
  • [DRILL-6912] - NPE when another drillbit is already running
  • [DRILL-6913] - Excessive error output
  • [DRILL-6915] - Unit test mysql-test-data.sql in contrib/jdbc-storage-plugin fails on newer MacOS
  • [DRILL-6916] - Fix extraneous "${project.basedir}/src/site/resources/repo/" directory appearance
  • [DRILL-6919] - Error: cannot find symbol in class ServerSocketUtil
  • [DRILL-6920] - Fix TestClient.testBasics() yarn test failure
  • [DRILL-6922] - QUERY-level options are shown on Profiles tab
  • [DRILL-6925] - Unable to generate Protobuf

New Feature

  • [DRILL-786] - Implement CROSS JOIN
  • [DRILL-2035] - Add ability to cancel multiple queries
  • [DRILL-5671] - Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
  • [DRILL-6179] - Added pcapng-format support
  • [DRILL-6381] - Add capability to do index based planning and execution
  • [DRILL-6385] - Support JPPD (Join Predicate Push Down)
  • [DRILL-6680] - Expose SHOW FILES command into INFORMATION_SCHEMA
  • [DRILL-6833] - MapRDB queries with Run Time Filters with row_key/Secondary Index Should Support Pushdown

Improvement

  • [DRILL-1248] - Add support for using aliases in group by
  • [DRILL-2374] - Sqlline prompt history needs improvements
  • [DRILL-3610] - TimestampAdd/Diff (SQL_TSI_) functions
  • [DRILL-3853] - Get off Sqlline fork
  • [DRILL-4456] - Hive translate function is not working
  • [DRILL-4697] - Editing of multi-line queries in sqlline
  • [DRILL-5735] - UI options grouping and filtering & Metrics hints
  • [DRILL-5796] - Filter pruning for multi rowgroup parquet file
  • [DRILL-6084] - Expose list of Drill functions for consumption by JavaScript libraries
  • [DRILL-6101] - Optimize Implicit Columns Processing
  • [DRILL-6492] - Make storage plugins names case insensitive
  • [DRILL-6544] - Allow timestamp / date / time formatting when displaying on Web UI
  • [DRILL-6611] - Add [meta]-[Enter] js handler for query form submission
  • [DRILL-6616] - Batch Processing for Lateral/Unnest
  • [DRILL-6629] - BitVector split and transfer does not work correctly for transfer length < 8
  • [DRILL-6633] - Replace usage of Guava classes by JDK ones
  • [DRILL-6640] - Drill takes long time in planning when there are large number of files in views/tables DFS parent directory
  • [DRILL-6644] - In Some Cases The HashJoin Memory Calculator Over Reserves Memory For The Probe Side During The Build Phase
  • [DRILL-6645] - Transform TopN in Lateral Unnest pipeline to Sort and Limit.
  • [DRILL-6650] - Remove Stray Semicolon in Printing Results Listener
  • [DRILL-6655] - Require Package Declaration In Checkstyle
  • [DRILL-6656] - Add Regex To Disallow Extra Semicolons In Imports
  • [DRILL-6662] - Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
  • [DRILL-6664] - Parquet reader should not allow batches with more than 64k rows
  • [DRILL-6668] - In Web Console, highlight options that are different from default values
  • [DRILL-6676] - Add Union, List and Repeated List types to Result Set Loader
  • [DRILL-6684] - Swap sys.options and sys.options_val tables
  • [DRILL-6687] - Improve RemovingRecordBatch to do transfer when all records needs to be copied
  • [DRILL-6689] - Include query user information to drillbit.log
  • [DRILL-6702] - OperatingSystemMXBean class cast exception when loaded under IBM JVM
  • [DRILL-6715] - Update descriptions for System Options table
  • [DRILL-6724] - Dump operator context to logs when error occurs during query execution
  • [DRILL-6731] - JPPD:Move aggregating the BF from the Foreman to the RuntimeFilter
  • [DRILL-6735] - Enhance the Hash-Join Operator to perform Semi-Join
  • [DRILL-6744] - Support filter push down for varchar / decimal data types
  • [DRILL-6755] - HashJoin should not build hash tables when probe side is empty.
  • [DRILL-6763] - Codegen optimization of SQL functions with constant values
  • [DRILL-6768] - Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when `drill.exec.functions.cast_empty_string_to_null` option is enabled
  • [DRILL-6791] - Merge scan projection framework into master
  • [DRILL-6792] - Find the right probe side fragment to any storage plugin
  • [DRILL-6817] - Update to_number function to be consistent with CAST function
  • [DRILL-6834] - Introduce option to disable result set for DDL queries for JDBC connection
  • [DRILL-6847] - Add Query Metadata to RESTful Interface
  • [DRILL-6864] - Root POM: Update the git-commit-id plugin
  • [DRILL-6898] - Web UI cannot be used without internet connection (jquery loaded from ajax.googleapis.com)

Test

  • [DRILL-6889] - Exclude Drill unit tests to avoid Travis timing out
  • [DRILL-6917] - Add test for DRILL-6912

Task

  • [DRILL-6422] - Update guava to 23.0 and shade it
  • [DRILL-6425] - Upgrade mapr release version
  • [DRILL-6473] - Update MapR Hive
  • [DRILL-6541] - Upgrade ZooKeeper patch version to 3.4.11 for mapr profile
  • [DRILL-6634] - Add udf module under contrib directory and move some udfs into it
  • [DRILL-6635] - PartitionLimit for Lateral/Unnest
  • [DRILL-6636] - Planner side changes to use PartitionLimitBatch in place of LimitBatch
  • [DRILL-6647] - Update Calcite version to 1.17.0
  • [DRILL-6691] - Unify checkstyle-config.xml files
  • [DRILL-6729] - Enable Unnest/Lateral Join feature by default
  • [DRILL-6751] - Upgrade Apache parent POM to version 21
  • [DRILL-6752] - Surround Drill quotes with double quotes
  • [DRILL-6765] - Exclude unused shaded guava classes from drill-jdbc-all jar
  • [DRILL-6790] - Updating SET and ALTER SYSTEM doc pages
  • [DRILL-6795] - Upgrade Janino compiler from 2.7.6 to 3.0.10
  • [DRILL-6818] - Add descriptions to secondary index options
  • [DRILL-6822] - Fix imports for DRILL-6381
  • [DRILL-6831] - Adding information about possible authentication settings in connection URL for mongodb
  • [DRILL-6866] - Upgrade to SqlLine 1.6.0
  • [DRILL-6868] - Upgrade Janino compiler to 3.0.11
  • [DRILL-6870] - Upgrade to ANTLR4
  • [DRILL-6895] - Warnings for license headers checks and wrong license headers in some files
← Apache Drill 1.16.0 Release NotesApache Drill 1.14.0 Release Notes →