ContextQMD
Back to Datafusion

31.0.0

dev/changelog/31.0.0.md

53.1.011.3 KB
Original Source
<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

31.0.0 (2023-09-08)

Full Changelog

Breaking changes:

  • Specialize Avg and Sum accumulators (#6842) #7358 (tustvold)
  • Use datum arithmetic scalar value #7375 (tustvold)

Implemented enhancements:

  • feat: array-empty #7313 (Weijun-H)
  • Support REPLACE SQL alias syntax #7368 (berkaysynnada)
  • feat: support primary key alternate syntax #7160 (parkma99)
  • docs: Add Expr library developer page #7359 (tshauck)
  • feat: support Binary for min/max #7397 (Weijun-H)
  • feat: Add memory pool configuration to datafusion-cli #7424 (Weijun-H)
  • Support Configuring Arrow RecordBatch Writers via SQL Statement Options #7390 (devinjdangelo)
  • Add ROLLUP and GROUPING SETS substrait support #7382 (nseekhao)
  • feat: Allow creating a ValuesExec from record batches #7444 (scsmithr)
  • minor: Add ARROW to CREATE EXTERNAL TABLE docs and add example of COMPRESSION TYPE #7489 (alamb)
  • Support Configuring Parquet Column Specific Options via SQL Statement Options #7466 (devinjdangelo)
  • Write Multiple Parquet Files in Parallel #7483 (devinjdangelo)
  • feat: explain with statistics #7459 (korowa)

Fixed bugs:

  • fix: union_distinct shouldn't remove child distinct #7346 (jackwener)
  • fix: inconsistent scalar types in DistinctArrayAggAccumulator state #7385 (korowa)
  • fix: incorrect nullability calculation of InListExpr #7496 (jonahgao)

Merged pull requests:

  • Remove redundant type check in Avg #7374 (viirya)
  • feat: array-empty #7313 (Weijun-H)
  • Minor: add WriteOp::name and DmlStatement::name #7329 (alamb)
  • Specialize Median Accumulator #7376 (tustvold)
  • Specialize Avg and Sum accumulators (#6842) #7358 (tustvold)
  • Change error type of invalid argument to PlanError rather than InternalError, remove misleading comments #7355 (alamb)
  • Implement array_pop_back function #7348 (tanruixiang)
  • Add exec_err! error macro #7361 (comphead)
  • Update sqlparser requirement from 0.36.1 to 0.37.0 #7387 (viirya)
  • DML documentation #7362 (spaydar)
  • Support REPLACE SQL alias syntax #7368 (berkaysynnada)
  • Bug-fix/next_value() of Min/Max Scalar Values #7384 (berkaysynnada)
  • Prepare 30.0.0 release #7372 (andygrove)
  • fix: union_distinct shouldn't remove child distinct #7346 (jackwener)
  • feat: support primary key alternate syntax #7160 (parkma99)
  • Merge hash table implementations and remove leftover utilities #7366 (metesynnada)
  • Minor: remove stray println! from array_expressions.rs #7389 (alamb)
  • Projection Order Propagation #7364 (berkaysynnada)
  • Document and scratch directory for sqllogictest and make test specific #7312 (alamb)
  • Minor: Move test for Distribution and Partitioning with code #7392 (alamb)
  • Minor: move datasource statistics code into its own module #7391 (alamb)
  • Use datum arithmetic scalar value #7375 (tustvold)
  • Fix IN expr for NaN #7378 (sarutak)
  • Unnecessary to list all files during partition pruning #7395 (smallzhongfeng)
  • Optimize Unnest and implement skip_nulls=true if specified #7371 (smiklos)
  • Docs: Add query syntax to COPY docs #7388 (alamb)
  • Clean up clippy for Rust 1.72 release #7399 (alamb)
  • fix: inconsistent scalar types in DistinctArrayAggAccumulator state #7385 (korowa)
  • Fix python CI #7416 (tustvold)
  • docs: Add Expr library developer page #7359 (tshauck)
  • Update ObjectStore 0.7.0 and Arrow 46.0.0 #7282 (tustvold)
  • Fix Decimal256 scalar display string in sqllogictest #7404 (viirya)
  • feat: support Binary for min/max #7397 (Weijun-H)
  • Make sqllogictest distinguish NaN from -NaN #7403 (sarutak)
  • Replace lazy_static with OnceLock #7409 (sarutak)
  • Minor: Remove the unreached simplification rule for 0 / 0 #7405 (jonahgao)
  • feat: Add memory pool configuration to datafusion-cli #7424 (Weijun-H)
  • Minor: Debug log when FairPool is created #7431 (alamb)
  • Support Configuring Arrow RecordBatch Writers via SQL Statement Options #7390 (devinjdangelo)
  • Add ROLLUP and GROUPING SETS substrait support #7382 (nseekhao)
  • Refactor sort_fuzz test to clarify what is covered #7439 (alamb)
  • Use DateTime::from_naive_utc_and_offset instead of DateTime::from_utc #7451 (sarutak)
  • Update substrait requirement from 0.12.0 to 0.13.1 #7443 (viirya)
  • [minior fix]: adjust the projection statistics #7428 (liukun4515)
  • Add new known users: Arroyo and Restate #7464 (jychen7)
  • ScalarFunctionExpr Maintaining Order #7417 (berkaysynnada)
  • Bug-fix/find_orderings_of_exprs #7457 (berkaysynnada)
  • Update prost-derive requirement from 0.11 to 0.12 #7468 (dependabot[bot])
  • Revert "Update prost-derive requirement from 0.11 to 0.12 (#7468)" #7476 (viirya)
  • Return error if spill file does not exist in ExternalSorter #7479 (viirya)
  • [minor fix]: Remove unused duplicate file_type.rs #7478 (sarutak)
  • Minor: more flexible pool size setting for datafusion-cli #7454 (yjshen)
  • Bump actions/checkout from 3 to 4 #7480 (dependabot[bot])
  • Support Write Options in DataFrame::write_* methods #7435 (devinjdangelo)
  • cp_solver, Duration vs Interval cases #7475 (berkaysynnada)
  • feat: Allow creating a ValuesExec from record batches #7444 (scsmithr)
  • Make LogicalPlan::with_new_exprs, deprecate from_plan #7396 (alamb)
  • refactor: change file type logic for create table #7477 (tshauck)
  • minor: do not fail analyzer if subquery plan contains extension #7455 (waynexia)
  • Make IN expr work with multiple items #7449 (sarutak)
  • Minor: Add doc comments and example for ScalarVaue::to_scalar #7491 (alamb)
  • minor: Add ARROW to CREATE EXTERNAL TABLE docs and add example of COMPRESSION TYPE #7489 (alamb)
  • Add backtrace to error messages #7434 (comphead)
  • Make sqllogictest platform-independent for the sign of NaN #7462 (sarutak)
  • Support Configuring Parquet Column Specific Options via SQL Statement Options #7466 (devinjdangelo)
  • Minor: improve error message #7498 (alamb)
  • Write Multiple Parquet Files in Parallel #7483 (devinjdangelo)
  • PrimitiveGroupsAccumulator should propagate timestamp timezone information properly #7494 (sunchao)
  • Minor: Add ScalarValue::data_type() for consistency with other APIs #7492 (alamb)
  • feat: explain with statistics #7459 (korowa)
  • fix: incorrect nullability calculation of InListExpr #7496 (jonahgao)