ContextQMD
Back to Datafusion

18.0.0

dev/changelog/18.0.0.md

53.1.016.5 KB
Original Source
<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

18.0.0 (2023-02-10)

Full Changelog

Breaking changes:

  • Use DataFusionError instead of ArrowError in SendableRecordBatchStream #5101 (comphead)
  • Update to arrow 32 and Switch to RawDecoder for JSON #5056 [sql] (tustvold)

Implemented enhancements:

  • DiskManager to create a spill folder if doesn't exist #5186
  • cast expression may cause duplicate column name error #5174
  • Add type coercion from Dictionary to string for regular expressions #5154
  • Unnecessary Filter on Parquet datasources #5149
  • [sqllogictest] Support pg_typeof for Postgres compatibility tests #5147
  • Supporting Grafana global variables #5144
  • add example for standalone DataFusion server which supports Arrow Flight SQL JDBC driver #5139
  • Support for InList in datafusion-substrait #5134
  • Pipeline file opening in FileStream #5129
  • Make parse_physical_expr public #5107
  • Use DataFusionError in SendableRecordBatchStream #5039
  • Interval coercion:date_bin('1 hour',...) does not work but date_bin(interval '1 hour', ... does #4853
  • Explain <query> should not fail if meeting errors when optimizing the query #4766
  • Add option to determine whether to convert identifiers #4551
  • Replace &Option<T> with Option<&T>. #4424
  • Error type in RecordBatchStream #4172
  • Support non-equi join (e.g. ON clause) in Dataframe API #1254
  • Allow ParquetExec to parallelize work based on row groups #137

Fixed bugs:

  • Confusing schema errors when using window partition #5229
  • Propagating empty_relation generate an illegal plan #5218
  • The test in_list_types_struct_literal fails when setting skip_failed_rules as false #5217
  • Placeholder values are not replaced in ScalarSubqueries #5215
  • Querying against delta lake table does not seem to work #5202
  • Arithmetic operation doesn't work with DictionaryArray #5193
  • simplify_expr() invoke nullable() exist bug #5191
  • CI is currently broken on git diff: Not a git repository #5180
  • write_csv/json/parquet isn't cancel safe #5178
  • no hyperlink to blaze-rs [doc: README-"Use Cases"] #5175
  • Arithmetic scalar operation doesn't work with DictionaryArray #5150
  • Sort operator disappear in physical_plan #5100
  • Window function error: InvalidArgumentError("number of columns(27) must match number of fields(35) in schema" #5090
  • INSERT statements without target column list are not working #5078
  • fix file stream time scanning metrics bug #5019
  • Date before 1678 causes panic #4875
  • Can not ORDER BY an aliased group column #4854
  • The filters expressions in TableScan may contain fields not included in schema. #4793
  • Comparing a Timestamp to a Date32 fails #4644
  • String --> TableReference parsing does not properly handle " and . #4532
  • can't compare NULL type with NULL type #4335
  • Add ambiguous check when generate selection plan #4196
  • Internal error in CAST from Timestamp[us] #3922
  • Run median expr on parquet file column got error #3805
  • aliasing a field renders it missing in the order by clause #669
  • Querying datetime data in DataFusion with an embedded timezone always fails #153

Documentation updates:

Closed issues:

  • Change coerced type for comparison between timestamp with date to timestamp #4761

Merged pull requests: