ContextQMD
Back to Datafusion

27.0.0

dev/changelog/27.0.0.md

53.1.021.2 KB
Original Source
<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

27.0.0 (2023-06-26)

Full Changelog

Breaking changes:

  • Remove avro_to_arrow::reader::Reader::next in favor of Iterator implementation. #6538 (LouisGariepy)
  • Add support for appending data to external tables - CSV #6526 (mustafasrepo)
  • Move physical_plan::file_format to datasource::plan #6516 (alamb)
  • Remove FromSlice in favor of From impl in upstream arrow-rs code #6587 (alamb)
  • Improve main api doc page, move avro_to_arrow to datasource #6564 (alamb)
  • Fix Clippy module inception (unwrap datasource::datasource and catalog::catalog #6640 (LouisGariepy)
  • refactor: unify generic expr rewrite functions into the datafusion_expr::expr_rewriter #6644 (r4ntix)
  • Move PhysicalPlanner to physical_planer module #6570 (alamb)
  • Update documentation for creating User Defined Aggregates (AggregateUDF) #6729 (alamb)
  • Support User Defined Window Functions #6703 (alamb)
  • Minor: Move PartitionStream to physical_plan #6756 (alamb)

Implemented enhancements:

  • feat: support type coercion in Parquet Reader #6458 (e1ijah1)
  • feat: New functions and operations for working with arrays #6384 (izveigor)
  • feat: DISTINCT bitwise and boolean aggregate functions #6581 (izveigor)
  • feat: make_array support empty arguments #6593 (parkma99)
  • feat: encapsulate physical optimizer rules into a struct #6645 (waynexia)
  • feat: new concatenation operator for working with arrays #6615 (izveigor)
  • feat: add -c option to pass the SQL query directly as an argument on datafusion-cli #6765 (r4ntix)

Fixed bugs:

  • fix: ignore panics if racing against catalog/schema changes #6536 (Weijun-H)
  • fix: type coercion support date - date #6578 (jackwener)
  • fix: avoid panic in list_files_for_scan #6605 (Folyd)
  • fix: analyze/optimize plan in CREATE TABLE AS SELECT #6610 (jackwener)
  • fix: remove type coercion of case expression in Expr::Schema #6614 (jackwener)
  • fix: correct test timestamp_add_interval_months #6622 (jackwener)
  • fix: fix more panics in ListingTable #6636 (Folyd)
  • fix: median with even number of Decimal128 not working #6634 (izveigor)
  • fix: port unstable subquery to sqllogicaltest #6659 (jackwener)
  • fix: correct wrong test #6667 (jackwener)
  • fix: from_plan shouldn't use original schema #6595 (jackwener)
  • fix: correct the error type #6712 (jackwener)
  • fix: parser for negative intervals #6698 (izveigor)

Documentation updates:

  • Minor: Fix doc for round function #6661 (viirya)
  • Docs: Improve documentation for struct function` #6754 (alamb)

Merged pull requests:

  • fix: ignore panics if racing against catalog/schema changes #6536 (Weijun-H)
  • Remove avro_to_arrow::reader::Reader::next in favor of Iterator implementation. #6538 (LouisGariepy)
  • Support ordering analysis with expressions (not just columns) by Replace OrderedColumn with PhysicalSortExpr #6501 (mustafasrepo)
  • Prepare for 26.0.0 release #6533 (andygrove)
  • fix Incorrect function-name matching with disabled enable_ident_normalization #6528 (parkma99)
  • Improve error messages with function name suggestion. #6520 (2010YOUY01)
  • Docs: add more PR guidance in contributing guide (smaller PRs) #6546 (alamb)
  • feat: support type coercion in Parquet Reader #6458 (e1ijah1)
  • Update to object_store 0.6 and arrow 41 #6374 (tustvold)
  • feat: New functions and operations for working with arrays #6384 (izveigor)
  • Add support for appending data to external tables - CSV #6526 (mustafasrepo)
  • [Minor] Update hashbrown to 0.14 #6562 (Dandandan)
  • refactor: use bitwise and boolean compute functions #6568 (izveigor)
  • Fix panic propagation in CoalescePartitions, consolidates panic propagation into RecordBatchReceiverStream #6507 (alamb)
  • Move physical_plan::file_format to datasource::plan #6516 (alamb)
  • refactor: remove type_coercion in PhysicalExpr. #6575 (jackwener)
  • Minor: remove tokio_stream dependency #6565 (alamb)
  • minor: remove useless mut and borrow() #6580 (jackwener)
  • Add tests for object_store builders of datafusion-cli #6576 (r4ntix)
  • Avoid per-batch field lookups in SchemaMapping #6563 (tustvold)
  • Move JoinType and JoinCondition to datafusion_common #6572 (alamb)
  • chore(deps): update substrait requirement from 0.10.0 to 0.11.0 #6579 (dependabot[bot])
  • refactor: bitwise kernel right and left shifts #6585 (izveigor)
  • fix: type coercion support date - date #6578 (jackwener)
  • make page filter public #6523 (jiacai2050)
  • Minor: Remove some use crate:: uses in physical_plan #6573 (alamb)
  • feat: DISTINCT bitwise and boolean aggregate functions #6581 (izveigor)
  • Make the struct function return the correct data type. #6594 (jiangzhx)
  • fix: avoid panic in list_files_for_scan #6605 (Folyd)
  • fix: analyze/optimize plan in CREATE TABLE AS SELECT #6610 (jackwener)
  • Minor: Add additional docstrings to Window function implementations #6592 (alamb)
  • Remove FromSlice in favor of From impl in upstream arrow-rs code #6587 (alamb)
  • [Minor] Cleanup tpch benchmark #6609 (Dandandan)
  • Revert "feat: Implement the bitwise_not in NotExpr (#5902)" #6599 (jackwener)
  • Port remaining tests in functions.rs to sqllogictest #6608 (jiangzhx)
  • fix: remove type coercion of case expression in Expr::Schema #6614 (jackwener)
  • Minor: use upstream dialect_from_str #6616 (alamb)
  • Minor: Move PlanType, StringifiedPlan and ToStringifiedPlan datafusion_common #6571 (alamb)
  • fix: correct test timestamp_add_interval_months #6622 (jackwener)
  • Impl Literal trait for NonZero* types #6627 (Folyd)
  • style: make clippy happy and remove redundant prefix #6624 (jackwener)
  • Substrait: Fix incorrect join key fields (indices) when same table is being used more than once #6135 (nseekhao)
  • Minor: Add debug logging for schema mismatch errors #6626 (alamb)
  • Minor: Move functionality into BuildInScalarFunction #6612 (alamb)
  • Add datafusion-cli tests to the CI Job #6600 (r4ntix)
  • Refactor joins test to sqllogic #6525 (aprimadi)
  • fix: fix more panics in ListingTable #6636 (Folyd)
  • fix: median with even number of Decimal128 not working #6634 (izveigor)
  • Unify formatting of both groups and files up to 5 elements #6637 (qrilka)
  • feat: make_array support empty arguments #6593 (parkma99)
  • Minor: cleanup the unnecessary CREATE TABLE aggregate_test_100 statement at aggregate.slt #6641 (jiangzhx)
  • chore(deps): update sqllogictest requirement from 0.13.2 to 0.14.0 #6646 (dependabot[bot])
  • Improve main api doc page, move avro_to_arrow to datasource #6564 (alamb)
  • Minor: Move include_rank into BuiltInWindowFunctionExpr #6620 (alamb)
  • Prioritize UDF over scalar built-in function in case of function name… #6601 (epsio-banay)
  • feat: encapsulate physical optimizer rules into a struct #6645 (waynexia)
  • Fix date_trunc signature #6632 (alamb)
  • Return correct scalar types for date_trunc #6638 (viirya)
  • Insert supports specifying column names in any order #6628 (jonahgao)
  • Fix Clippy module inception (unwrap datasource::datasource and catalog::catalog #6640 (LouisGariepy)
  • Add hash support for PhysicalExpr and PhysicalSortExpr #6625 (mustafasrepo)
  • Port tests in joins.rs to sqllogictes #6642 (jiangzhx)
  • Minor: Add test for date_trunc schema on scalars #6655 (alamb)
  • Simplify and encapsulate window function state management #6621 (alamb)
  • Minor: Move get_equal_orderings into BuiltInWindowFunctionExpr, remove BuiltInWindowFunctionExpr::as_any #6619 (alamb)
  • minor: use sql to setup test data for joins.slt rather than rust #6656 (alamb)
  • Support wider range of Subquery, handle the Count bug #6457 (mingmwang)
  • fix: port unstable subquery to sqllogicaltest #6659 (jackwener)
  • Minor: Fix doc for round function #6661 (viirya)
  • refactor: unify generic expr rewrite functions into the datafusion_expr::expr_rewriter #6644 (r4ntix)
  • Minor: add test cases for coercion bitwise shifts #6651 (izveigor)
  • refactor: unify replace count(*) analyzer by removing it in sql crate #6660 (jackwener)
  • Combine evaluate_stateful and evaluate_inside_range #6665 (mustafasrepo)
  • Support internal cast for BuiltinScalarFunction::MakeArray #6607 (jayzhan211)
  • minor: use sql to setup test data for aggregate.slt rather than rust #6664 (jiangzhx)
  • Minor: Add tests for User Defined Aggregate functions #6669 (alamb)
  • fix: correct wrong test #6667 (jackwener)
  • fix: from_plan shouldn't use original schema #6595 (jackwener)
  • feat: new concatenation operator for working with arrays #6615 (izveigor)
  • Minor: Add more doc strings to WindowExpr #6663 (alamb)
  • minor: with_new_inputs replace from_plan #6680 (jackwener)
  • Docs: Update roadmap to point at EPIC's, clarify project goals #6639 (alamb)
  • Disable incremental compilation on CI #6688 (alamb)
  • Allow AggregateUDF to define retractable batch , implement sliding window functions #6671 (alamb)
  • Minor: Update user guide #6692 (comphead)
  • Minor: consolidate repartition test into sql_integration to save builder space and build time #6685 (alamb)
  • Minor: combine statistics, filter_pushdown and custom_sources provider tests together to reduce CI disk space #6683 (alamb)
  • Move PhysicalPlanner to physical_planer module #6570 (alamb)
  • Rename integration tests to match crate they are defined in #6687 (alamb)
  • Minor: combine fuzz tests into a single binary to save builder space and build time #6684 (alamb)
  • Minor: consolidate datafusion_substrait tests into substrait_integration to save builder space and build time #6685 #6686 (alamb)
  • removed self.all_values.len() from inside reserve #6689 (BryanEmond)
  • Replace supports_bounded_execution with supports_retract_batch #6695 (mustafasrepo)
  • Move dataframe and dataframe_functon into core_integration test binary #6697 (alamb)
  • refactor: fix clippy allow too many arguments #6705 (aprimadi)
  • Fix documentation typo #6704 (aprimadi)
  • fix: correct the error type #6712 (jackwener)
  • Port test in subqueries.rs from rust to sqllogictest #6675 (jiangzhx)
  • Improve performance/memory usage of HashJoin datastructure (5-15% improvement on selected TPC-H queries) #6679 (Dandandan)
  • refactor: alias() should skip add alias for Expr::Sort #6707 (jackwener)
  • chore(deps): update strum/strum_macros requirement from 0.24 to 0.25 #6717 (jackwener)
  • Move alias generator to per-query execution props #6706 (aprimadi)
  • fix: parser for negative intervals #6698 (izveigor)
  • Minor: Improve UX for setting ExecutionProps::query_execution_start_time #6719 (alamb)
  • add Eq and PartialEq to ListingTableUrl #6725 (fsdvh)
  • Support Expr::InList to Substrait::RexType #6604 (jayzhan211)
  • MINOR: Add maintains input order flag to CoalesceBatches #6730 (mustafasrepo)
  • Minor: Update copyight date on website #6727 (alamb)
  • Display all partitions and files in EXPLAIN VERBOSE #6711 (qrilka)
  • Update arrow, arrow-flight and parquet to 42.0.0 #6702 (alamb)
  • Move PartitionEvaluator and window_state structures to datafusion_expr crate #6690 (alamb)
  • Hash Join Vectorized collision checking #6724 (Dandandan)
  • Return null for date_trunc(null) instead of panic #6723 (BryanEmond)
  • derive(Debug) for Expr #6708 (parkma99)
  • refactor: extract merge_projection common function. #6735 (jackwener)
  • Fix up some DataFusionError::Internal errors with correct type #6721 (alamb)
  • Minor: remove some uses of unwrap #6738 (alamb)
  • Minor: remove dead code with decimal datatypes from in_list #6737 (izveigor)
  • Update documentation for creating User Defined Aggregates (AggregateUDF) #6729 (alamb)
  • Support User Defined Window Functions #6703 (alamb)
  • MINOR: Aggregate ordering substrait support #6745 (mustafasrepo)
  • chore(deps): update itertools requirement from 0.10 to 0.11 #6752 (jackwener)
  • refactor: move some code in physical_plan/common.rs before tests module #6749 (aprimadi)
  • Add support for order-sensitive aggregation for multipartitions #6734 (mustafasrepo)
  • Update sqlparser-rs to version 0.35.0 #6753 (alamb)
  • Docs: Update SQL status page #6736 (alamb)
  • fix typo #6761 (Weijun-H)
  • Minor: Move PartitionStream to physical_plan #6756 (alamb)
  • Docs: Improve documentation for struct function` #6754 (alamb)
  • add UT to verify the fix on "issues/6606" #6762 (mingmwang)
  • Re-export modules individually to fix rustdocs #6757 (alamb)
  • Order Preserving RepartitionExec Implementation #6742 (mustafasrepo)
  • feat: add -c option to pass the SQL query directly as an argument on datafusion-cli #6765 (r4ntix)