ContextQMD
Back to Datafusion

28.0.0

dev/changelog/28.0.0.md

53.1.019.3 KB
Original Source
<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

28.0.0 (2023-07-21)

Full Changelog

Breaking changes:

  • Cleanup type coercion (#3419) #6778 (tustvold)
  • refactor: encapsulate Alias as a struct #6795 (jackwener)
  • Set DisplayAs to be a supertrait of ExecutionPlan #6835 (qrilka)
  • [MINOR] Remove unnecessary api from MemTable #6861 (metesynnada)
  • refactor: Merge Expr::Like and Expr::ILike #7007 (waynexia)

Implemented enhancements:

  • feat: array_contains #6618 (izveigor)
  • feat: support NULL in array functions #6662 (izveigor)
  • feat: implement posgres style encode/decode #6821 (ozgrakkurt)
  • feat: column support for array_append, array_prepend, array_position and array_positions #6805 (izveigor)
  • feat: preserve metadata for Field and Schema in proto #6865 (jonahgao)
  • feat: Add graphviz display format for execution plan. #6726 (liurenjie1024)
  • feat: implement substrait join filter support #6868 (nseekhao)
  • feat: column support for array_dims, array_ndims, cardinality and array_length #6864 (izveigor)
  • feat: support for NestedLoopJoinExec in datafusion-proto #6902 (r4ntix)
  • feat: add round trip test of physical plan in tpch unit tests #6918 (r4ntix)
  • feat: implement substrait for LIKE/ILIKE expr #6840 (waynexia)
  • feat: array functions treat an array as an element #6986 (izveigor)

Fixed bugs:

  • fix: incorrect nullability of between expr #6786 (jonahgao)
  • fix: incorrect nullability of InList expr #6799 (jonahgao)
  • fix: from_plan generate Agg can be with different schema. #6820 (jackwener)
  • fix: incorrect nullability of Like expressions #6829 (jonahgao)
  • fix: incorrect simplification of case expr #7006 (jonahgao)
  • fix: array_concat with arrays with different dimensions, add _list* aliases for _array* functions #7008 (izveigor)

Documentation updates:

  • docs: Add encode and decode to the user guide #6856 (alamb)

Merged pull requests:

  • chore(deps): update indexmap requirement from 1.9.2 to 2.0.0 #6766 (dependabot[bot])
  • Support IsDistinctFrom and IsNotDistinctFrom on interval types #6776 (joroKr21)
  • Protect main branch #6775 (tustvold)
  • Prepare 27.0.0 release #6773 (andygrove)
  • Support hex string literal #6767 (ShiKaiWi)
  • feat: array_contains #6618 (izveigor)
  • Make 'date_trunc' returns the same type as its input #6654 (Weijun-H)
  • Fix inserting into a table with non-nullable columns #6722 (jonahgao)
  • Cleanup type coercion (#3419) #6778 (tustvold)
  • Properly project grouping set expressions #6777 (fsdvh)
  • Minor: Simplify date_trunc code and add comments #6783 (alamb)
  • Minor: Add array / array sqllogic tests for array_contains #6771 (alamb)
  • Minor: Make date_trunc code easier to understand #6789 (alamb)
  • feat: support NULL in array functions #6662 (izveigor)
  • fix: incorrect nullability of between expr #6786 (jonahgao)
  • Use checked division kernel #6792 (tustvold)
  • Minor: add sqllogictests for binary data type #6770 (alamb)
  • refactor: encapsulate Alias as a struct #6795 (jackwener)
  • chore(deps): bump actions/labeler from 4.1.0 to 4.2.0 #6803 (dependabot[bot])
  • Consistently coerce dictionaries for arithmetic #6785 (tustvold)
  • Implement serialization for UDWF and UDAF in plan protobuf #6769 (parkma99)
  • fix: incorrect nullability of InList expr #6799 (jonahgao)
  • Fix timestamp_add_interval_months to pass any date #6815 (jayzhan211)
  • Minor: Log TPCH benchmark results #6813 (alamb)
  • Refactor Decimal128 averaging code to be vectorizable (and easier to read) #6810 (alamb)
  • Minor: Encapsulate return_type and signature in AggregateFunction and WindowFunction #6748 (alamb)
  • fix: from_plan generate Agg can be with different schema. #6820 (jackwener)
  • [MINOR] Improve performance of create_hashes #6816 (Dandandan)
  • Add fetch to SortPreservingMergeExec and SortPreservingMergeStream #6811 (Dandandan)
  • chore(deps): update substrait requirement from 0.11.0 to 0.12.0 #6825 (dependabot[bot])
  • Upgrade arrow 43 #6812 (tustvold)
  • Fix cargo build warning #6831 (viirya)
  • Simplify IsUnkown and IsNotUnkown expression #6830 (jonahgao)
  • fix: incorrect nullability of Like expressions #6829 (jonahgao)
  • Minor: Add one more assert to hash_array_primitive #6834 (alamb)
  • revert #6595 #6820 #6827 (jackwener)
  • Add Duration to ScalarValue #6838 (tustvold)
  • Replace AbortOnDrop / AbortDropOnMany with tokio JoinSet #6750 (aprimadi)
  • Add clickbench queries to sqllogictest coverage #6836 (alamb)
  • feat: implement posgres style encode/decode #6821 (ozgrakkurt)
  • chore(deps): update rstest requirement from 0.17.0 to 0.18.0 #6847 (dependabot[bot])
  • [minior] support serde for some function #6846 (liukun4515)
  • Support fixed_size_list for make_array #6759 (jayzhan211)
  • Improve median performance. #6837 (vincev)
  • Mismatch in MemTable of Select Into when projecting on aggregate window functions #6566 (berkaysynnada)
  • feat: column support for array_append, array_prepend, array_position and array_positions #6805 (izveigor)
  • MINOR: Fix ordering of the aggregate_source_with_order table #6852 (mustafasrepo)
  • Return error when internal multiplication overflowing in decimal division kernel #6833 (viirya)
  • Deprecate ScalarValue::and, ScalarValue::or (#6842) #6844 (tustvold)
  • chore(deps): update bigdecimal requirement from 0.3.0 to 0.4.0 #6848 (dependabot[bot])
  • feat: preserve metadata for Field and Schema in proto #6865 (jonahgao)
  • Set DisplayAs to be a supertrait of ExecutionPlan #6835 (qrilka)
  • [MINOR] Remove unnecessary api from MemTable #6861 (metesynnada)
  • Adjustment of HashJoinExec APIs to Preserve Probe Side Order #6858 (metesynnada)
  • [MINOR] Adding order into StreamingTableExec #6860 (metesynnada)
  • Docs: try and clarify what PartitionEvaluator functions are called #6869 (alamb)
  • docs: Add encode and decode to the user guide #6856 (alamb)
  • Fix build on main due to logical conflict #6875 (alamb)
  • feat: Add graphviz display format for execution plan. #6726 (liurenjie1024)
  • Fix (another) logical conflict #6882 (alamb)
  • Minor: Consolidate display related traits #6883 (alamb)
  • test: parquet use the byte array as the physical type to store decimal #6851 (smallzhongfeng)
  • Make streaming_merge public #6874 (kazuyukitanimura)
  • Performance: Use a specialized sum accumulator for retractable aggregregates #6888 (alamb)
  • Support array concatenation for arrays with different dimensions #6872 (jayzhan211)
  • feat: implement substrait join filter support #6868 (nseekhao)
  • feat: column support for array_dims, array_ndims, cardinality and array_length #6864 (izveigor)
  • Add FixedSizeBinary support to binary_op_dyn_scalar #6891 (maxburke)
  • Minor: deleted duplicated substrait integration test #6894 (alamb)
  • Minor: add test cases with columns for math expressions #6787 (izveigor)
  • Minor: reduce redundant code #6901 (smallzhongfeng)
  • Minor: Add some more doc comments to BoundedAggregateStream #6881 (alamb)
  • feat: support for NestedLoopJoinExec in datafusion-proto #6902 (r4ntix)
  • Fix make_array null handling, update tests #6900 (alamb)
  • chore(deps): bump actions/labeler from 4.2.0 to 4.3.0 #6911 (dependabot[bot])
  • Minor: Add TPCH scale factor 10 to bench.sh, use 10 iteration #6893 (alamb)
  • Minor: Add output to aggregrate_fuzz.rs on failure #6905 (alamb)
  • allow window UDF to return null #6915 (mhilton)
  • Minor: Add factory method to PartitionedFile to create File Scan #6909 (comphead)
  • [minor]fix doc to remove duplicate content #6923 (liukun4515)
  • Revert "chore(deps): update bigdecimal requirement from 0.3.0 to 0.4.0 (#6848)" #6896 (alamb)
  • [Minor] Make FileScanConfig::project pub #6931 (Dandandan)
  • feat: add round trip test of physical plan in tpch unit tests #6918 (r4ntix)
  • Minor: Use thiserror to implement the From trait for DFSqlLogicTestError #6924 (jonahgao)
  • parallel csv scan #6801 (2010YOUY01)
  • Add additional test coverage for aggregaes using dates/times/timestamps/decimals #6939 (alamb)
  • Replace repartition execs with sort preserving repartition execs #6921 (mertak)
  • Vectorized hash grouping #6904 (alamb)
  • Fix incorrect results in BitAnd GroupsAccumulator #6957 (alamb)
  • Fixes for clippy 1.71 #6959 (alamb)
  • Improve unnest_column performance #6903 (vincev)
  • Pass schema_infer_max_records to JsonFormat. #6945 (vincev)
  • deps: bump sqllogictest to 0.15.0 #6941 (jonahgao)
  • Preserve field metadata across expressions in logical plans #6920 (dexterduck)
  • Support equality and comparison between interval arrays and scalars #6948 (joroKr21)
  • chore(deps): update bigdecimal requirement from 0.3.0 to 0.4.1 #6946 (dependabot[bot])
  • feat: implement substrait for LIKE/ILIKE expr #6840 (waynexia)
  • Minor: Add comments about initial value for BitAnd accumulator #6964 (alamb)
  • [Functions] Support Arithmetic function COT() #6925 (Syleechan)
  • Minor: remove duplication in Min/Max accumulator #6960 (alamb)
  • [MINOR]Add new tests #6953 (mustafasrepo)
  • Column support for array concat #6879 (jayzhan211)
  • Minor: Add FixedSizeBinaryTest #6895 (alamb)
  • [MINOR] Remove update state api from PartitionEvaluator #6966 (mustafasrepo)
  • Fix required partitioning of Single aggregation mode #6950 (Dandandan)
  • [MINOR] Remove global sort rule from planner #6965 (mustafasrepo)
  • Column support for array_to_string #6940 (jayzhan211)
  • chore: fix format #6991 (Weijun-H)
  • Extend Ordering Equivalence Support #6956 (mustafasrepo)
  • chore: break earlier in macro contains! #6989 (Weijun-H)
  • fix: incorrect simplification of case expr #7006 (jonahgao)
  • Minor: Add String/Binary aggregate tests #6962 (alamb)
  • [MINOR] Supporting repartition joins conf in SHJ #6998 (metesynnada)
  • [MINOR] Code refactor on hash join utils #6999 (metesynnada)
  • feat: array functions treat an array as an element #6986 (izveigor)
  • [MINOR] Moving some test utils from EnsureSorting to test_utils #7009 (metesynnada)
  • MINOR: Bug fix, Use correct ordering equivalence when window expr contains partition by #7011 (mustafasrepo)
  • refactor: Merge Expr::Like and Expr::ILike #7007 (waynexia)
  • Docs: Add docs to RepartitionExec and architecture guide #7003 (alamb)
  • Consolidate BoundedAggregateStream #6932 (alamb)
  • Minor: Improve aggregate test coverage more #6952 (alamb)
  • Don't store hashes in GroupOrdering #7029 (tustvold)
  • Extract GroupValues (#6969) #7016 (tustvold)
  • Refactor AnalysisContext and statistics() of FilterExec #6982 (berkaysynnada)
  • Fix datafusion-cli/Dockerfile to build successfully #7031 (sarutak)
  • functions: support trunc() function with one or two args #6942 (Syleechan)
  • Move the column aliases below the SubqueryAlias #7035 (jonahgao)
  • fix: array_concat with arrays with different dimensions, add _list* aliases for _array* functions #7008 (izveigor)
  • Add support for ClickBench in bench.sh #7005 (alamb)
  • Remove RowAccumulators and datafusion-row #6968 (alamb)
  • Decimal256 coercion #7034 (jdye64)
  • Double RawTable on grow instead of triple #7041 (tustvold)
  • Specialize single column primitive group values #7043 (tustvold)