ContextQMD
Back to Datafusion

Apache DataFusion 54.0.0 Changelog

dev/changelog/54.0.0.md

55.0.098.9 KB
Original Source
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

Apache DataFusion 54.0.0 Changelog

This release consists of 740 commits from 139 contributors. See credits at the end of this changelog for more information.

See the upgrade guide for information on how to upgrade from previous versions.

Breaking changes:

  • Add ExecutionPlan::apply_expressions() #20337 (LiaCastaneda)
  • Add Field to Expr::Cast -- allow logical expressions to express a cast to an extension type #18136 (paleolimbot)
  • feat: parse JsonAccess as a binary operator, add Operator::Colon #20628 (Samyak2)
  • Wrap Arc to Statistics for partition_statistics API #20570 (xudong963)
  • Replace ahash with foldhash for faster hashing in datafusion-common #20958 (Dandandan)
  • fix: arrays_zip/list_zip allow single array argument #21047 (hsiang-c)
  • Remove file prefetching from FileStream #20916 (Dandandan)
  • Remove as_any from scalar UDF trait definition #20812 (timsaucer)
  • Provide session to the udtf call #20222 (askalt)
  • chore: remove as_any from aggregate and window functions #21209 (timsaucer)
  • chore: remove as_any from ExecutionPlan #21263 (timsaucer)
  • fix: Prefer numeric in type coercion for comparisons #20426 (neilconway)
  • refactor(pruning): remove column param from PruningStatistics::row_counts #21369 (adriangb)
  • Remove CastColumnExpr and custom_file_casts example; unify on field-aware CastExpr #21563 (kosiew)
  • perf: Optimize NULL handling in StringViewArrayBuilder #21538 (neilconway)
  • Remove as_any on the PhysicalExpr trait #21573 (timsaucer)
  • Remove trait function as_any from datafusion-datasource #21576 (timsaucer)
  • feat: change approx percentile/median UDFs to return floats #21074 (theirix)
  • chore: Rename concat-specific string builders, make pub(crate) #21695 (neilconway)
  • perf: Implement physical execution of uncorrelated scalar subqueries #21240 (neilconway)
  • Add lambda support and array_transform udf #21679 (gstvg)
  • perf: strength reduce hash partition modulo (up to 1.16x faster) #21900 (Dandandan)
  • feat: Improve InListExpr types, flatten dict haystacks and validate in try_new_from_array #21402 (buraksenn)
  • feat: type-keyed extensions map for PartitionedFile #21993 (adriangb)
  • Add support for lambda column capture #21323 (gstvg)
  • feat: Add Protobuf support for Explain node #21994 (danielhumanmod)
  • deprecate: mark Statistics V2 framework (PR #14699) as deprecated #22071 (alamb)
  • feat: impl Any for MemoryPool #21803 (haohuaijin)
  • Add metrics to FFI_ExecutionPlan #22136 (mailmindlin)
  • fix(aggregate): show aliased expr in explain #21739 (kumarUjjawal)
  • proto: serialize dynamic filters on Sort, Aggregate, HashJoin plan nodes #22011 (jayshrivastava)
  • Add exact HigherOrderSignature #22326 (LiaCastaneda)
  • Add a memory bound FileStatisticsCache for the Listing Table #20047 (mkleen)
  • Add configurable UNION DISTINCT to FILTER rewrite optimization #21075 (xiedeyantu)
  • minor: make HigherOrderSignature less error-prone #22106 (gstvg)
  • Expose ExecutionPlan statistics across the FFI boundary #22157 (mailmindlin)
  • feat: optional timezone for coerce_int96 #22318 (andygrove)

Performance related:

  • perf: Optimize array_to_string to avoid a copy #20639 (neilconway)
  • perf: Apply logical regexp optimizations to Utf8View and LargeUtf8 inputs #20581 (petern48)
  • perf: Optimize array_concat using MutableArrayData #20620 (neilconway)
  • perf: Optimize to_char to allocate less, fix NULL handling #20635 (neilconway)
  • Eliminate deterministic group by keys with deterministic transformations #20706 (Dandandan)
  • perf: short-circuit and collect_bool for IN list with column references #20694 (zhangxffff)
  • perf: sort replace free()->try_grow() pattern with try_resize() to reduce memory pool interactions #20729 (mbutrovich)
  • perf: Optimize set operations to avoid RowConverter deserialization overhead #20623 (neilconway)
  • perf: Use batched row conversion for array_has_any, array_has_all #20588 (neilconway)
  • perf: Optimize array set ops on sliced arrays #20693 (neilconway)
  • perf: Optimize comparison on nested types #20716 (neilconway)
  • perf: Optimize array_positions() for scalar needle #20770 (neilconway)
  • perf: Optimize approx_distinct() for string, binary inputs #21037 (neilconway)
  • perf: Optimize approx_distinct for inline Utf8View #21064 (neilconway)
  • perf: Optimize strpos() for scalar needle, plus optimize UTF-8 codepath #20754 (neilconway)
  • perf: Optimize lpad(), rpad() for scalar args #20657 (neilconway)
  • perf: add in-place fast path for ScalarValue::add #20959 (kumarUjjawal)
  • perf: Optimize array_sort() #21083 (neilconway)
  • Super fast extended tests and improved planning speed linux #21084 (blaginin)
  • Add a builder to SimplifyContext to avoid allocating default values #21092 (AdamGS)
  • Avoid creating new RecordBatches to simplify expressions #20534 (alamb)
  • perf: optimize scatter with type-specific specialization #20498 (CuteChuanChuan)
  • perf: Optimize array_min, array_max for arrays of primitive types #21101 (neilconway)
  • perf: optimize map validation for common key types #20805 (lyne7-sc)
  • perf: specialized SemiAntiSortMergeJoinStream #20806 (mbutrovich)
  • Improvement: keep order-preserving repartitions for streaming aggregates #21107 (xudong963)
  • perf: Add support for GroupsAccumulator to string_agg #21154 (neilconway)
  • perf: Optimize split_part, support Utf8View #21119 (neilconway)
  • perf: sort-merge join (SMJ) batch deferred filtering and move mark joins to bitwise stream. Near-unique LEFT and FULL SMJ 20-50x faster #21184 (mbutrovich)
  • perf: Optimize string_to_array for scalar args #21131 (neilconway)
  • Misc minor optimizations to query optimizer performance #21128 (AdamGS)
  • ensure dynamic filters are correctly pushed down through aggregations #21059 (jayshrivastava)
  • perf: Merge Precision in-place #21219 (AdamGS)
  • feat: support GroupsAccumulator for first_value and last_value with string/binary types #21090 (UBarney)
  • perf: Optimize split_part for scalar args #21238 (neilconway)
  • perf: optimize object store requests when reading JSON #20823 (ariel-miculas)
  • perf: Optimize split_part for Utf8View #21420 (neilconway)
  • Eliminate outer joins with empty relations via null-padded projection #21321 (SubhamSinghal)
  • Optimize regexp_replace by stripping trailing .* from anchored patterns. 2.4x improvement (ClickBench Q28) #21379 (Dandandan)
  • perf: use DynComparator in sort-merge join (SMJ), microbenchmark queries up to 12% faster, TPC-H overall ~5% faster #21484 (mbutrovich)
  • perf: Optimize NULL handling in substr #21519 (neilconway)
  • perf: replace SMJ's join_filter_not_matched_map HashMap with Vec<FilterState> #21517 (mbutrovich)
  • perf: Optimize NULL handling in find_in_set #21464 (neilconway)
  • perf: Optimize NULL handling in lcm, gcd #21468 (neilconway)
  • perf: Optimize NULL handling in arrays_zip #21475 (neilconway)
  • perf: Optimize NULL handling in array_remove #21532 (neilconway)
  • perf: Optimize NULL handling in array_slice #21482 (neilconway)
  • perf: Optimize NULL handling in some datetime functions #21477 (neilconway)
  • perf: Optimize NULL handling in array_has #21471 (neilconway)
  • perf: Optimize Utf8View string concat #21535 (neilconway)
  • Conditionally build page pruning predicates #21480 (fpetkovski)
  • perf: add fast path for uniform fill values in array_resize #20617 (lyne7-sc)
  • perf : Optimize count distinct using bitmaps instead of hashsets for smaller datatypes #21456 (coderfender)
  • perf: Optimize left, right to reduce copying #21442 (neilconway)
  • perf: Optimize substr for Utf8, LargeUtf8 #21366 (neilconway)
  • feat: Optimize ORDER BY by Pruning Functionally Redundant Sort Keys #21362 (xiedeyantu)
  • perf: Optimize logical optimizer's OptimizeProjections pass #21726 (neilconway)
  • perf: Optimize DFSchema::qualified_name #21722 (neilconway)
  • perf: Tweak vec capacity in project_statistics #21734 (neilconway)
  • perf: Reduce Box and Arc allocation churn during tree rewriting #21749 (neilconway)
  • perf: Implement groups accumulator count distinct primitive types #21561 (coderfender)
  • perf: Optimize approx count distinct using bitmaps instead of HLL for smaller int datatypes #21453 (coderfender)
  • perf: Optimize lower, upper for sliced arrays #21814 (neilconway)
  • perf: Add bulk NULL-aware string builders, use in lower and upper #21789 (neilconway)
  • perf: Use bulk-NULL builder in uuid #21845 (neilconway)
  • Skip map_expressions rebuild for Extension nodes with empty expressions #21701 (zhuqi-lucas)
  • Refactor InListExpr into static-filter modules #21649 (geoffreyclaude)
  • perf: Use bulk-NULL string builder in initcap #21863 (neilconway)
  • perf: Use bulk-NULL builder in chr #21847 (neilconway)
  • perf: implement convert_to_state for SparkAvg #21548 (azhangd)
  • perf: optimise first_value, last_value aggregate function #21383 (theirix)
  • perf(spark): use 256-entry byte-pair table in hex encoding #21836 (Scolliq)
  • perf: Optimize substr_index to use bulk-NULL string builder #21877 (neilconway)
  • perf: Use bulk-NULL builder in replace #21849 (neilconway)
  • Add SQL based benchmarking harness, port tpch to use framework #21707 (Omega359)
  • perf: Add BulkNullStringArrayBuilder trait, use in repeat #21854 (neilconway)
  • perf: optimize retract_batch for median and percentile_cont #21894 (lyne7-sc)
  • perf: Optimize reverse using bulk-NULL string builders #21991 (neilconway)
  • perf: Optimize lower, upper for ASCII inputs #21980 (neilconway)
  • perf: Cast entire Date32 array to Date64 on 1st failure #21948 (huymq1710)
  • perf: Use NullBuffer::union_many #22070 (neilconway)
  • perf: improve Int64 generate_series and range performance #21891 (lyne7-sc)
  • perf: batch contiguous extend calls in array_replace #22119 (lyne7-sc)
  • perf: Add append_with to string builders, use in replace #22029 (neilconway)
  • perf: reuse mask in truncate_list_nulls and avoid counting all true bits #22158 (rluvaton)
  • Skip RowFilter and page pruning for fully matched row groups #21637 (xudong963)
  • perf: bypass values.value(i) for inline strings in ArrowBytesViewMap #22172 (RyanJamesStewart)
  • perf: Elimiate SortExec on generate_series() #22238 (2010YOUY01)
  • perf: coalesce batches before sending to distributor channels in RepartitionExec #22010 (gabotechs)
  • Resolve MIN/MAX from Parquet metadata for Single-mode aggregates and CAST projections #21651 (Dandandan)
  • Compact more aggressively in TopK based upon memory usage #20381 (cetra3)

Implemented enhancements:

  • feat: support nanosecond date_part #20674 (mhilton)
  • feat: Support Spark array_contains builtin function #20685 (comphead)
  • feat: Integrate CastColumnExpr into PhysicalExprAdapter #20269 (kumarUjjawal)
  • feat: partition_statistics() for HashJoinExec #20711 (jonathanc-n)
  • feat: make DefaultLogicalExtensionCodec support serialisation of buil… #20638 (Acfboy)
  • feat: correct struct column names for arrays_zip return type #20886 (comphead)
  • feat: Reduce allocations for aggregating Statistics #20768 (jonathanc-n)
  • feat: add custom_string_literal_override to unparser Dialect trait #20590 (goldmedal)
  • feat: Extract NDV (distinct_count) statistics from Parquet metadata #19957 (asolimando)
  • feat: support repartitioning of FFI execution plans #20449 (timsaucer)
  • feat: create a datafusion-example for in-memory file format #20394 (kumarUjjawal)
  • feat: implement PhysicalOptimizerRule in FFI crate #20451 (timsaucer)
  • feat(metric): Add output skewness metric to detect skewed plans easier #21211 (2010YOUY01)
  • feat: add sort pushdown benchmark and SLT tests #21213 (zhuqi-lucas)
  • feat(sql): unparse array_has as ANY for Postgres #20654 (vimeh)
  • feat: feature-gate sqllogictests datafusion-substrait behind optional 'substrait' feature #21268 (zhuqi-lucas)
  • feat: generate reversed-name data for sort pushdown benchmark #21266 (zhuqi-lucas)
  • feat: Complete basic LATERAL JOIN functionality #21202 (neilconway)
  • feat: Use NDV for equality filter selectivity calculation #20789 (jonathanc-n)
  • feat: make BatchPartitioner::partition_iter public #21341 (hcrosse)
  • feat: spark compatible float to timestamp cast with ANSI support #21212 (coderfender)
  • feat(spark): Adds spark round function #21062 (SubhamSinghal)
  • feat: make DataFrame::create_physical_plan take &self instead of self #20562 (xanderbailey)
  • feat: add support for parquet content defined chunking options #21110 (kszucs)
  • feat: sort file groups by statistics during sort pushdown (Sort pushdown phase 2) #21182 (zhuqi-lucas)
  • feat: Set NDV to Exact(1) for numeric equality filter predicates #21077 (asolimando)
  • feat: make sort pushdown BufferExec capacity configurable, default 1GB #21426 (zhuqi-lucas)
  • feat: Propagate orderings through struct-producing projections #21218 (rkrishn7)
  • feat: add cast_to_type UDF for type-based casting #21322 (adriangb)
  • feat: Add pluggable StatisticsRegistry for operator-level statistics propagation #21483 (asolimando)
  • feat: Add Hash trait to Aggregate enums #21569 (rluvaton)
  • feat(substrait): support Placeholder <-> DynamicParameter in Substrait producer/consumer #20977 (bvolpato)
  • feat: add with_metadata scalar UDF to attach Arrow field metadata #21509 (adriangb)
  • feat: Additional Canonical Extension Types #21291 (tschwarzinger)
  • feat: Add memory-limited execution for NestedLoopJoinExec #21448 (viirya)
  • feat(stats): cap NDV at row count in statistics estimation #21081 (asolimando)
  • feat: support array_compact builtin function #21522 (comphead)
  • feat: add a config to disable subquery_sort_elimination #21614 (haohuaijin)
  • feat: extend single ndv optimization to non-arithmetic supporting types for equality predicates #21473 (buraksenn)
  • feat: extend interval analysis support for temporal types #21520 (buraksenn)
  • feat: add sort_pushdown_inexact benchmark for RG reorder #21674 (zhuqi-lucas)
  • feat: support '>', '<', '>=', '<=', '<>' in all operator #21416 (buraksenn)
  • feat: Add support for LEFT JOIN LATERAL #21352 (neilconway)
  • feat: Expose used MemoryPool details in ResourcesExhausted error messages #20387 (erenavsarogullari)
  • feat: estimate cardinality for semi and anti-joins using distinct counts #20904 (buraksenn)
  • feat: support ListView and LargeListView in ScalarValue #21669 (Jefffrey)
  • feat: add cosine_distance scalar function #21542 (crm26)
  • feat: remove __unnest_placeholder from struct unnest projection #21725 (akoshchiy)
  • feat(unparser): Keep inner join Filter → TableScan predicates to WHERE instead of moving to JOIN ON #21694 (sgrebnov)
  • feat: minor lambda perf improvements #21896 (comphead)
  • feat: automatically cast ListView to List for UDFs #21855 (Jefffrey)
  • feat: support binary arguments for StringConcat operator #21883 (theirix)
  • feat: add inner_product scalar function #21861 (crm26)
  • feat: Support RIGHT/FULL joins in NLJ memory-limited execution #21833 (viirya)
  • feat: Improved multiple column aggregation performance by using bitmasks rather than Vec<bool> #21886 (huymq1710)
  • feat: Making From<ConfigFileDecryptionProperties/ConfigFileEncryptionProperties> conversions fallible with TryFrom #21985 (Soham-Bhattacharjee-work)
  • feat: support spark compatible floor function #21933 (athlcode)
  • feat: fix NTILE distribution logic #22051 (comphead)
  • feat: implement retract_batch for array_agg sliding window support #22015 (SubhamSinghal)
  • feat: Upgrade to sqlparser-rs 0.62.0 #22069 (andygrove)
  • feat: fix windows frame positive/neg overflows #22140 (comphead)
  • feat: fix AVG sliding windows wrong results with NULLs #22139 (comphead)
  • feat: fix windows decimal casting frame #22174 (comphead)
  • feat: eliminate GlobalLimitExec when input statistics prove limit is already satisfied #22150 (xiedeyantu)
  • feat: globally reorder files and row groups by statistics for TopK queries #21956 (zhuqi-lucas)
  • feat: Restore nullability when consuming substrait fields #22105 (neilconway)
  • feat: add array_normalize scalar function #22013 (crm26)
  • feat: add Spark-compatible xxhash64 function #21967 (andygrove)

Fixed bugs:

  • fix: make the sql feature truly optional #20625 (linhr)
  • fix: use try_shrink instead of shrink in try_resize #20424 (ariel-miculas)
  • fix: Provide more generic API for the capacity limit parsing #20372 (erenavsarogullari)
  • fix: Fix bug in array_has scalar path with sliced arrays #20677 (neilconway)
  • fix: HashJoin panic with String dictionary keys (don't flatten keys) #20505 (alamb)
  • fix: Return probe_side.len() for RightMark/Anti count(*) queries #20710 (jonathanc-n)
  • fix: preserve None projection semantics across FFI boundary in ForeignTableProvider::scan #20393 (Kontinuation)
  • fix(spark): handle divide-by-zero in Spark mod/pmod with ANSI mode support #20461 (davidlghellin)
  • fix: sqllogictest cannot convert <subquery> to Substrait #19739 (kumarUjjawal)
  • fix: interval analysis error when have two filterexec that inner filter proves zero selectivity #20743 (haohuaijin)
  • fix: SanityCheckPlan error with window functions and NVL filter #20231 (EeshanBembi)
  • fix: Avoid unnecessary type casts in concat_ws #20436 (neilconway)
  • fix: Remove !=0 check from supports_collect_by_thresholds #20730 (jonathanc-n)
  • fix: do not recompute hash join exec properties if not required #20900 (askalt)
  • fix: Optimize !~ '.*' case to col IS NULL AND Boolean(NULL) instead of Eq "" #20702 (petern48)
  • fix: Track metrics in hash joins with empty build sides #20810 (nuno-faria)
  • fix: dfbench respects DATAFUSION_RUNTIME_MEMORY_LIMIT env var #20631 (adriangb)
  • fix(spark): return input string for PATH/FILE on schemeless URLs in parse_url #20506 (davidlghellin)
  • fix: InList Dictionary filter pushdown type mismatch #20962 (erratic-pattern)
  • fix: Run release verification with --profile=ci #20987 (alamb)
  • fix: move overflow guard before dense ratio in hash join to prevent overflows #20998 (buraksenn)
  • fix: improve GroupOrdering docs #20994 (alamb)
  • fix: update clickbench expected plan for NDV-aware optimization #21050 (asolimando)
  • fix: use datafusion_expr instead of datafusion crate in spark #21043 (davidlghellin)
  • Fix CTE reference resolution slt tests #21049 (jonahgao)
  • fix: validate wrapped negation during type coercion #20965 (myandpr)
  • fix(sql): handle GROUP BY ALL with aliased aggregates #20943 (kumarUjjawal)
  • fix: string_to_array('', delim) returns empty array for PostgreSQL compatibility #21104 (dd-david-levin)
  • Fix push_down_filter for children with non-empty fetch fields #21057 (shivbhatia10)
  • fix(stats): widen sum_value integer arithmetic to SUM-compatible types #20865 (kumarUjjawal)
  • fix: skip empty metadata in intersect_metadata_for_union to prevent s… #21127 (RafaelHerrero)
  • fix: Df int timestamp cast fix failing CI #21163 (coderfender)
  • fix(unparser): Fix BigQuery timestamp literal format in SQL unparsing #21103 (sgrebnov)
  • fix: propagate errors for unsupported table function arguments instead of silently dropping them #21135 (buraksenn)
  • fix: Fix main compilation failure #21242 (2010YOUY01)
  • fix: Revert "Fix/support duplicate column names #6543 (#21126)" #21254 (mbutrovich)
  • fix: Fix three bugs in query decorrelation #21208 (neilconway)
  • fix: date overflow panic #21233 (haohuaijin)
  • fix: SELECT * EXCLUDE(...) silently returns empty rows when all columns are excluded #21259 (xiedeyantu)
  • fix(unparser): use to_rfc3339 for default TIMESTAMPTZ formatting #21295 (sgrebnov)
  • fix: use spill writer's schema instead of the first batch schema for spill files #21293 (gruuya)
  • fix: binary string concat #20787 (theirix)
  • fix(sql): fix a bug when planning semi- or antijoins #20990 (aalexandrov)
  • fix(datasource): keep stats absent when collect_stats is false #21149 (kumarUjjawal)
  • fix: preserve source field metadata in TryCast expressions #21390 (adriangb)
  • fix: skips projection pruning for whole subtree #20545 (Acfboy)
  • fix: preserve subquery structure when unparsing SubqueryAlias over Ag… #21099 (yonatan-sevenai)
  • fix: FilterExec should drop projection when apply projection pushdown #21460 (haohuaijin)
  • fix: preserve duplicate GROUPING SETS rows #21058 (xiedeyantu)
  • fix: apply the left side schema on the right side in set expressions #21052 (gruuya)
  • fix: Use codepoints in lpad, rpad, translate #21405 (neilconway)
  • fix: PostgreSQL dialect can not support tinyint type #21445 (xiedeyantu)
  • fix: DataFusion benchmark panicked: failed to cast '2013-07-01' to UInt16 #21498 (xiedeyantu)
  • fix(sql): return planner error for malformed typed literals #21454 (officialasishkumar)
  • fix: Preserve quoted mixed-case identifiers in the pivot_unpivot example #21432 (niebayes)
  • fix(spark): array_repeat returns repeated NULLs instead of NULL when element is NULL #21558 (buraksenn)
  • fix: grouping with alias #21438 (timsaucer)
  • fix(spark): mod/pmod returns NULL instead of NaN for float division by zero #21557 (buraksenn)
  • fix: LazyMemoryExec should produce independent streams per execute() #21565 (viirya)
  • fix: json scan performance on local files #21478 (ariel-miculas)
  • fix(benchmarks): correct TPC-H benchmark SQL #21615 (kumarUjjawal)
  • fix: suppress nondeterministic metrics in agg_dyn_e2e sqllogictest #21657 (mbutrovich)
  • fix: Fix compilation error on main #21664 (2010YOUY01)
  • fix: median retract logic for sliding window frames #21300 (lyne7-sc)
  • fix: Fix Spark slice function Null type to GenericListArray casting issue #20469 (erenavsarogullari)
  • fix: Remove nested async block causing Stacked Borrows violation in PushDecoderStreamState #21663 (mbutrovich)
  • fix: impl handle_child_pushdown_result for SortExec #21527 (haohuaijin)
  • fix: SortMergeJoin full outer join incorrectly matches rows when filter evaluates to NULL #21660 (mbutrovich)
  • fix: try again to fix Miri in ParquetOpener #21680 (mbutrovich)
  • fix: optimize_projections failure after mark joins created by EXISTS OR EXISTS #21265 (buraksenn)
  • fix: import from datafusion_expr in make_valid_utf8 #21687 (hcrosse)
  • fix: linearized operands in physical binaryexpr protobuf to avoid recursion limit #21031 (haohuaijin)
  • fix: remove unnecessary as_any() to fix compilation error #21693 (Jefffrey)
  • fix: Prevent CLI crash on wide tables #21721 (Geethapranay1)
  • fix(unparser): make BigQueryDialect more robust #21296 (sgrebnov)
  • fix: insert placeholder type inference showing wrong type when there is function wrapped placeholder (unknown type) #20744 (buraksenn)
  • fix: array_concat widens container variant for mixed List/LargeList inputs #21704 (hcrosse)
  • fix: Fix local datafusion-cli test failure #21761 (2010YOUY01)
  • fix: Validate spill read schema #21738 (2010YOUY01)
  • fix: improve sort pushdown benchmark data and add DESC LIMIT queries #21711 (zhuqi-lucas)
  • fix: rebind RecursiveQueryExec batches to the declared output schema #21770 (adriangb)
  • fix: Enable arrow-ipc/zstd in datasource-arrow to make test_spill_compression pass in every config #21504 (AdamGS)
  • fix: Do not highlight the CLI hint directly #21858 (nuno-faria)
  • fix: fix elapsed_compute metric in ParquetSink to report encoding time only #21825 (fred1268)
  • fix: grouping separator for float and decimal #20268 (Druva-D)
  • fix: Fix .gitignore in benchmarks/ #21954 (2010YOUY01)
  • fix(proto): correctly serialize FilterExec empty projection #21885 (Adez017)
  • fix: Make conversion from FileDecryptionProperties to ConfigFileDecryptionProperties fallible #21603 (adamreeve)
  • fix: Avoid unnecessary input repartitioning with ScalarSubqueryExec #21986 (neilconway)
  • fix: error on CREATE EXTERNAL TABLE with no files and no explicit schema #21965 (adriangb)
  • fix: median returns Float64 for integer inputs to avoid truncation #21988 (CuteChuanChuan)
  • fix: Correct the number of pruned/matched Parquet pages #22031 (nuno-faria)
  • fix: use datafusion_expr instead of datafusion crate #22052 (hsiang-c)
  • fix(spark): align parse_url empty FILE path #21969 (kumarUjjawal)
  • fix: drop input plan early in CoalescePartitionsExec #22017 (Samyak2)
  • fix: track join_arrays memory in reservation after SMJ spill #21962 (SubhamSinghal)
  • fix: Avoid overlay panic on valid Unicode input, Postgres compatibility #22046 (neilconway)
  • fix: Panic in Spark's format_string for illegal characters #22077 (neilconway)
  • fix: Incorrect behavior for FILTER on NULLs #22068 (neilconway)
  • fix: coerce operand types in Interval mul/div/intersect/union/contains #22027 (adriangb)
  • fix(bench): avoid OOM in array_replace bench #22120 (kumarUjjawal)
  • fix: Nested self-referential CASE chains should not cause exponential hashing work during physical planning. #22175 (avantgardnerio)
  • fix: preserve Inexact precision in Statistics #22146 (timsaucer)
  • fix: Handle EXECUTE without statement name #22204 (Dandandan)
  • fix(sql): reject duplicate unqualified names in CTAS, CREATE VIEW, and SELECT INTO #22290 (kumarUjjawal)
  • fix: reduce memory allocation overhead during partial aggregation ear… #22165 (ariel-miculas)
  • fix: Fix bug with structurally equal correlated subqueries #22313 (neilconway)
  • fix: return error instead of capacity overflow panic in generate_series #22323 (sweb)
  • fix: simplifier on leaf nodes returns null #22368 (timsaucer)

Documentation updates:

  • Update DataFusion meetups page on docs #20629 (alamb)
  • docs: Update datafusion-cli doc for top-memory-consumers config #20390 (erenavsarogullari)
  • [main] Update version to 52.2.0 #20573 (alamb)
  • Update releases links with releases in 2025-2026 #20630 (alamb)
  • doc: Add more context to Precision #20713 (jonathanc-n)
  • Minor: Add comment explaining rationale to avoid dependencies on functions #20667 (alamb)
  • Hash join buffering on probe side #19761 (gabotechs)
  • Copy limits before repartitions #20736 (avantgardnerio)
  • Allow SQL TypePlanner to plan SQL types as extension types #20676 (paleolimbot)
  • doc: Add documentation for pushing limit into plan #20271 (2010YOUY01)
  • [main] Bump to 52.3.0 and changelog (#20790) #20849 (alamb)
  • refactor: Improve SessionContext::parse_duration API #20816 (erenavsarogullari)
  • docs: in release email, be specific about changelog location #20975 (kevinjqliu)
  • optimizer: Add configuration to disable join reordering #21072 (2010YOUY01)
  • docs: Improve getting started and testing guides for humans and agents #20970 (alamb)
  • docs: clarify NULL handling for array_remove functions (#21014) #21018 (Xavrir)
  • chore: Add substr() benchmarks, refactor #20803 (neilconway)
  • docs: Document the TableProvider evaluation order for filter, limit and projection #21091 (alamb)
  • Add arrow_try_cast UDF #21130 (adriangb)
  • docs: Add explicit fmt and clippy commands to AGENTS.md #21171 (zhuqi-lucas)
  • docs: add KalamDB to known users #21181 (jamals86)
  • [main] Update version to 53.0.0 and bring changelog #21189 (alamb)
  • Migrate Avro reader to arrow-avro and remove internal conversion code #17861 (getChan)
  • Add metric category filtering for EXPLAIN ANALYZE #21160 (adriangb)
  • docs: Add RESET Command Documentation #21245 (erenavsarogullari)
  • chore: fix upgrade guide link for object_store release notes #21283 (haohuaijin)
  • doc: Add documentation explaining the behavior of null values ​​in struct comparisons #21226 (xiedeyantu)
  • [docs] Add weekly sync details to contributor communication guide #21298 (alamb)
  • [docs] add sql example to timestamp/datetime docs for time zone #21082 (buraksenn)
  • Update documentation with recent blogs and events #21462 (alamb)
  • Update 53 upgrade guide to note release, other changes #21449 (alamb)
  • docs: Incorporate writing table provider blog post to user documentation #21398 (buraksenn)
  • remove as_any from TableProvider, SchemaProvider, CatalogProvider, and CatalogProviderList #21346 (timsaucer)
  • port 52.5.0 changelog to main #21553 (alamb)
  • Add arrow_field(expr) scalar UDF #21389 (adriangb)
  • Reorder cargo publish commands by dependency #21552 (alamb)
  • chore(deps): update jinja2 requirement from <4,>=3.1 to >=3.1.6,<4 in /docs #21606 (dependabot[bot])
  • chore(deps): update pydata-sphinx-theme requirement from <1,>=0.16 to >=0.17.0,<1 in /docs #21609 (dependabot[bot])
  • Add release management page to the documentation #21001 (alamb)
  • Perf: Window topn optimisation #21479 (SubhamSinghal)
  • chore(deps): update setuptools requirement from <83,>=82 to >=82.0.1,<83 in /docs #21607 (dependabot[bot])
  • chore(deps): update maturin requirement from <2,>=1.11 to >=1.13.1,<2 in /docs #21608 (dependabot[bot])
  • docs: Update map_extract examples #21360 (nuno-faria)
  • docs: add April 2026 readings and meetup links #21644 (alamb)
  • chore: backport version from branch-53, update some dependencies #21708 (comphead)
  • chore: add array_remove_* NULL handling changes to Upgrade Guide #21769 (comphead)
  • docs: fix some comments on query_planning example #21783 (jotare)
  • docs: fix typos in documentation #21875 (jx2lee)
  • docs: refresh CLI usage output in the user guide #21874 (jx2lee)
  • docs: clarify ExecutionProps and TaskContext docs #21872 (alamb)
  • chore: add internal markdown link check #21831 (Geethapranay1)
  • Update documentation for PhysicalExpr::evaluate_bounds #21879 (alamb)
  • chore(deps): update pydata-sphinx-theme requirement from <1,>=0.17.0 to >=0.17.1,<1 in /docs #21889 (dependabot[bot])
  • docs(optimizer): add generated optimizer rules reference #21824 (kumarUjjawal)
  • add any_match higher-order function #21903 (LiaCastaneda)
  • docs: update commiter list #21978 (coderfender)
  • chore: update PMC/committer list #21989 (comphead)
  • Support '0' value for parse_capacity_limit() #22014 (mkleen)
  • docs: add llms.txt ecosystem hub at site root #22003 (timsaucer)
  • chore(deps): update maturin requirement from <2,>=1.13.1 to >=1.13.3,<2 in /docs #22127 (dependabot[bot])
  • fix date_part('isodow') #22116 (sdf-jkl)
  • docs: updating arrays_zip output field naming #22133 (timsaucer)
  • Add rand() alias for random() #22147 (xiedeyantu)
  • chore: Update Rust toolchain to 1.95 #22177 (Dandandan)
  • docs: add DataFusion Java to subproject listings #22149 (andygrove)
  • Fix: deadlink in "Concepts, Reading, Events" page to DataFusion blog #22325 (JarroVGIT)
  • fixing factorial negative values #22278 (raushanprabhakar1)
  • docs(optimizer): Fix PushDownFilter doc typos. #22320 (JSOD11)
  • minor: add higher-order function methods to SessionContext #21950 (gstvg)
  • Add higher-order functions changes to upgrade guide #22107 (gstvg)
  • chore(deps): update myst-parser requirement from <6,>=5 to >=5.1.0,<6 in /docs #22378 (dependabot[bot])
  • feat(functions-nested): add array_filter higher-order function #21895 (ologlogn)
  • Add SQL as a category in breaking API change policy #22179 (alamb)
  • [branch-54] Bump to version 54.0.0 #22396 (mbutrovich)
  • [branch-54] Revert "Add ExecutionPlan::apply_expressions() (#20337)" (#22437) #22445 (alamb)
  • [branch-54] Gate new ScalarSubqueryExec node behind session property (#22530) #22690 (LiaCastaneda)

Other:

  • Add metrics for parquet sink #20307 (xudong963)
  • Extend dynamic filter to joins that preserve probe side ON #20447 (helgikrs)
  • Improve sqllogicteset speed by creating only a single large file rather than 2 #20586 (Tim-53)
  • cli: Fix datafusion-cli hint edge cases #20609 (comphead)
  • Speedup sqllogictests by running long running tests first #20576 (alamb)
  • Fix custom metric display #20643 (gabotechs)
  • refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit #20375 (erenavsarogullari)
  • more families for the CI #20663 (blaginin)
  • CI: Add CodeQL workflow for GitHub Actions security scanning #20636 (kevinjqliu)
  • chore(deps): bump astral-sh/setup-uv from 7.3.0 to 7.3.1 #20660 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.68.8 to 2.68.16 #20661 (dependabot[bot])
  • Improve formatting of datatypes #20605 (emilk)
  • Add explain plans for ClickBench queries #20666 (alamb)
  • Add files_processed and files_scanned metrics to FileStreamMetrics #20592 (adriangb)
  • Speedup push_down_filter_regression.slt by using uncompressed parquet #20652 (alamb)
  • Implement cardinality_effect for window execs and UnionExec #20321 (getChan)
  • ci: Harden labeler workflow, remove unnecessary checkout from pull_request_target job #20637 (kevinjqliu)
  • Add tests for sqllogictest prioritization #20656 (alamb)
  • correct parquet leaf index mapping when schema contains struct cols #20698 (friendlymatthew)
  • Reattach parquet metadata cache after deserializing in datafusion-proto #20574 (nathanb9)
  • Wire up with_new_state with DataSource #20718 (gabotechs)
  • chore: Enable assigning_clones clippy lint #20670 (neilconway)
  • FFI_TableOptions are using default values only #20721 (timsaucer)
  • Improve documentation for AggregateUdfImpl::simplify and WindowUDFImpl::simplify #20712 (alamb)
  • Fix test that's broken on Windows due to naive path handling #20692 (Rafferty97)
  • Fix DELETE/UPDATE filter extraction when predicates are pushed down into TableScan #19884 (kosiew)
  • use linker optimization for extended sqllogictests #20740 (blaginin)
  • Push even local limits past windows #20752 (avantgardnerio)
  • Add case-heavy LEFT JOIN benchmark and debug timing/logging for PushDownFilter hot paths #20664 (kosiew)
  • Fix repartition from dropping data when spilling #20672 (xanderbailey)
  • test: Add datafusion-cli fair and unbounded memory-pool test coverage #20565 (erenavsarogullari)
  • ser/de fetch in FilterExec #20738 (haohuaijin)
  • Add tests for simplifying multiple aggregate expressions #20723 (alamb)
  • Update reverse UDF to emit utf8view when input is utf8view #20604 (Omega359)
  • Make lower and upper emit Utf8View for Utf8View input #20616 (kumarUjjawal)
  • Fix FilterExec converting Absent column stats to Exact(NULL) #20391 (fwojciec)
  • Clean up date_part preimage implementation #20350 (sdf-jkl)
  • Make Physical CastExpr Field-aware and unify cast semantics across physical expressions #20814 (kosiew)
  • Pass ConfigOptions to scalar UDFs via FFI #20454 (timsaucer)
  • [datafusion-cli] Replace mutex with AtomicU64 for stream duration tracking in instrumentedObjectStore #20802 (buraksenn)
  • Make translate emit Utf8View for Utf8View input #20624 (shivaaang)
  • Allow filters on struct fields to be pushed down into Parquet scan #20822 (friendlymatthew)
  • Used constant with mapping instead of write! to display scalar value bytes #20719 (buraksenn)
  • chore(deps): bump taiki-e/install-action from 2.68.16 to 2.68.25 #20842 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.32.5 to 4.32.6 #20843 (dependabot[bot])
  • chore: Ignore RUSTSEC-2024-0421 #20850 (comphead)
  • chore(deps): bump quinn-proto from 0.11.13 to 0.11.14 #20859 (dependabot[bot])
  • Use ParquetPushDecoder in ParquetOpener #20839 (Dandandan)
  • [Minor] Remove redundant ProjectionExec nodes in sort-based plans #20780 (Dandandan)
  • impl ser/de for preserve_order in RepartitionExec #20798 (haohuaijin)
  • Fix FileStream scanning_total to include sync next-file open time #20627 (RatulDawar)
  • chore: Ignore RUSTSEC-2024-0014 #20862 (comphead)
  • chore: clean up dependencies #20861 (comphead)
  • Add benchmark for struct field filter pushdown in Parquet #20829 (friendlymatthew)
  • Add Null Type Coercions for Placeholders #20543 (cetra3)
  • Minor: Deprecate unused PartitionedFileStream #20869 (alamb)
  • chore(deps): bump substrait from 0.62 to 0.63.0 #20876 (benbellick)
  • [Minor] propagate distinct_count as inexact through unions #20846 (buraksenn)
  • try to remove redundant alias in expression rewriter and select #20867 (buraksenn)
  • Fix duplicate group keys after hash aggregation spill (#20724) #20858 (gboucher90)
  • Include .proto files in datafusion-proto-common distribution #20921 (haohuaijin)
  • Check sqllogictests for any dangling config settings (#17914) #20838 (cj-zhukov)
  • Add support for ListView in unnest #20760 (brancz)
  • Project only accessed struct leaves in Parquet row filter pushdown #20854 (friendlymatthew)
  • minor: Move PreparedAccessPlan to same module as ParquetAccessPlan #20929 (alamb)
  • chore(deps): bump pyjwt from 2.11.0 to 2.12.0 #20938 (dependabot[bot])
  • Rewrite SUM(expr + scalar) --> SUM(expr) + scalar*COUNT(expr) #20749 (alamb)
  • Add AGENTS.md / CLAUDE.md #20939 (Dandandan)
  • Support columns_sorted in row_filters #20497 (sdf-jkl)
  • Add --simulate-latency / SIMULATE_LATENCY option to dfbench / ./bench.sh #20954 (Dandandan)
  • Minor: make signatures of SessionContext::register_* methods consistent #20873 (alexandreyc)
  • test: add reproducer for Dictionary InList pushdown type mismatch (#2… #20960 (erratic-pattern)
  • Extract shared ParquetReadPlan for leaf column resolution #20913 (friendlymatthew)
  • chore: Remove usage of paste crate #20946 (coderfender)
  • Use exact distinct_count from statistics if exists for COUNT(DISTINCT column)) calculations #20845 (buraksenn)
  • thin-ci #20972 (blaginin)
  • chore(deps): bump lz4_flex from 0.12.0 to 0.12.1 #20973 (dependabot[bot])
  • Fix decimal log precision for non-power values #20433 (kumarUjjawal)
  • chore(deps): bump Swatinem/rust-cache from 2.8.2 to 2.9.1 #20979 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.68.25 to 2.68.34 #20983 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.32.6 to 4.33.0 #20982 (dependabot[bot])
  • chore(deps): bump astral-sh/setup-uv from 7.3.1 to 7.6.0 #20981 (dependabot[bot])
  • chore(deps): bump runs-on/action from 2.0.3 to 2.1.0 #20980 (dependabot[bot])
  • [Minor] Update Cargo.lock, Fix Tokio minor breaking change #20978 (Dandandan)
  • chore(deps): Revert "chore(deps): bump runs-on/action from 2.0.3 to 2.1.0 (#20980)" #21002 (mbutrovich)
  • bug: fix array_remove_* with NULLS #21013 (comphead)
  • Simplify logic for memory pressure partial emit from ordered group by #20559 (alamb)
  • Fix memory reservation starvation in sort-merge #20642 (xudong963)
  • infra: automatically delete branch on pr merge #21033 (kevinjqliu)
  • Add support for nested lists in substrait consumer #20953 (alexanderbianchi)
  • build: update Rust toolchain version to 1.94.0 #21045 (dariocurr)
  • chore: Cleanup fully-qualified ScalarFunctionArgs #20804 (neilconway)
  • Support '>', '<', '>=', '<=', '<>' in any operator #20830 (buraksenn)
  • keep fetch when merge FilterExec in FilterPushdown #21070 (haohuaijin)
  • Fix Subtraction overflow in max_distinct_count when hash join has a pushed-down limit #20799 (KARTIK64-rgb)
  • Restore Sort unparser guard for correct ORDER BY placement #20658 (krinart)
  • chore(deps): bump rustls-webpki from 0.103.9 to 0.103.10 #21089 (dependabot[bot])
  • chore: Remove duplicate imports in test code #21061 (neilconway)
  • test: update sqllogictest expectation for negation type coercion #21102 (myandpr)
  • fix[physical-expr-adapter]: support casting structs nested inside complex types #20907 (asubiotto)
  • Fix index panic in unparser with mismatched stacked projections #21094 (friendlymatthew)
  • chore: Fix all sqllogictest dangling configs #21108 (2010YOUY01)
  • Preserve SPM when parent maintains input order #21097 (rkrishn7)
  • chore: update testcontainers and astral-tokio-tar for cargo audit #21114 (getChan)
  • Spark soundex function implementation #20725 (kazantsev-maksim)
  • chore(deps): bump env_logger from 0.11.9 to 0.11.10 in the all-other-cargo-deps group across 1 directory #21136 (dependabot[bot])
  • Fix elapsed_compute metric for Parquet DataSourceExec #20767 (ernestprovo23)
  • chore(deps): bump taiki-e/install-action from 2.68.34 to 2.69.7 #21133 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.33.0 to 4.34.1 #21132 (dependabot[bot])
  • Update to arrow/parquet 58.1.0 #21044 (alamb)
  • Simplify sqllogictest timing summary to boolean flag and remove top-N modes #20598 (kosiew)
  • Substrait join consumer should not merge nullability of join keys #21121 (hareshkh)
  • Enable debug assertions in CI. #20832 (stuhood)
  • chore(deps): bump requests from 2.32.5 to 2.33.0 #21153 (dependabot[bot])
  • feat : support spark compatible int to timestamp cast #20555 (coderfender)
  • [Minor]: support window functions in order by expressions #20963 (buraksenn)
  • chore: Optimize schema rewriter usages #21158 (comphead)
  • Add benchmarks for Parquet struct leaf-level projection pruning #21180 (friendlymatthew)
  • chore: re-export projection in datafusion::datasource #21185 (rluvaton)
  • test: add SMJ benchmarks from #21184 #21188 (mbutrovich)
  • Fix sort merge interleave overflow #20922 (xudong963)
  • Reduce parquet struct projection benchmark data volume #21187 (friendlymatthew)
  • Minor: compute qualify window expressions only when QUALIFY clause is present #21173 (buraksenn)
  • fix[physical-plan/aggregates]: fix grouping by Ree<Dict> #21195 (asubiotto)
  • [main] add 52.4.0 changelog #21053 (alamb)
  • Use leaf level ProjectionMask for parquet projections #20925 (friendlymatthew)
  • test: scale remaining sort-merge join (SMJ) benchmark queries #21200 (mbutrovich)
  • Fix: MemTable LIMIT ignored with reordered projections #21177 (RamakrishnaChilaka)
  • No cargo test for sort_mem_validation #21222 (blaginin)
  • Fix/support duplicate column names #6543 #21126 (RafaelHerrero)
  • Use spot instances for extended tests #21221 (blaginin)
  • chore: Cleanup Cargo profiles #21214 (neilconway)
  • chore(benchmark): Fix/update compile profile benchmark #21223 (2010YOUY01)
  • Basic Extension Type Registry Implementation #20312 (tschwarzinger)
  • chore(deps): bump serialize-javascript, terser-webpack-plugin and copy-webpack-plugin in /datafusion/wasmtest/datafusion-wasm-app #21235 (dependabot[bot])
  • chore(deps-dev): bump node-forge from 1.3.2 to 1.4.0 in /datafusion/wasmtest/datafusion-wasm-app #21225 (dependabot[bot])
  • chore(deps): bump cryptography from 46.0.5 to 46.0.6 #21224 (dependabot[bot])
  • Fix FilterExec tree render missing fetch display #21230 (zhuqi-lucas)
  • ci: use ubuntu-slim runner for lightweight CI jobs #21252 (CuteChuanChuan)
  • kill check_run_id and pr_number from extended tests #21228 (blaginin)
  • [Minor] add non topk benchmarks for utf8/utf8view string aggregates #21073 (buraksenn)
  • ci: Add datafusion/sql as a folder to trigger extended tests for on changes #21255 (mbutrovich)
  • Misc minor optimization in the Physical Optimizer #21216 (AdamGS)
  • chore: Replace TryInto impl by TryFrom #21203 (Tpt)
  • Refactor parquet datasource into an explicit state machine #21190 (alamb)
  • Add flat vs. struct field projection benchmarks #21257 (friendlymatthew)
  • Refactor: expose predicate constant inference from physical-expr #21167 (xudong963)
  • Add end-to-end Parquet tests for List and LargeList struct schema evolution #20840 (kosiew)
  • chore(deps): bump taiki-e/install-action from 2.69.7 to 2.70.3 #21271 (dependabot[bot])
  • chore(deps): bump rustyline from 17.0.2 to 18.0.0 #21276 (dependabot[bot])
  • chore(deps): bump ctor from 0.6.3 to 0.8.0 #21282 (dependabot[bot])
  • chore(deps): bump snmalloc-rs from 0.3.8 to 0.7.4 #21280 (dependabot[bot])
  • chore(deps): bump sha1 from 0.10.6 to 0.11.0 #21277 (dependabot[bot])
  • chore(deps): bump astral-sh/setup-uv from 7.6.0 to 8.0.0 #21272 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.34.1 to 4.35.1 #21273 (dependabot[bot])
  • chore(deps): bump pygments from 2.19.2 to 2.20.0 #21256 (dependabot[bot])
  • feat(memory_pool): add TrackConsumersPool::metrics() to expose cons… #21147 (bert-beyondloops)
  • Update repeat UDF to emit utf8view when input is utf8view #20645 (Omega359)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 7 updates #21274 (dependabot[bot])
  • chore(deps): bump runs-on/action from 2.0.3 to 2.1.0 #21134 (dependabot[bot])
  • chore: add .claude/settings.local.json to .gitignore #21312 (jonahgao)
  • Add FileStreamBuilder for creating FileStreams #21261 (alamb)
  • refactor: Split Parquet BloomFilter CPU and IO into separate states #21285 (alamb)
  • chore(deps): bump object_store from 0.13.1 to 0.13.2 #21275 (dependabot[bot])
  • Merge queue: make dev checks required + add .asf.yaml validation #21239 (blaginin)
  • Adds INList and Between expr to skip outer join #21303 (SubhamSinghal)
  • No merge group for rust.yml yet #21343 (blaginin)
  • Disallow order by within ordered-set aggregate functions argument lists #20421 (cj-zhukov)
  • chore: Fix clippy and CI #21287 (comphead)
  • Split FileStreamMetrics into its own module #21340 (alamb)
  • Skip probe-side consumption when hash join build side is empty #21068 (kosiew)
  • Use ParquetMetaDataPushDecoder instead of ParquetMetaDataReader #21357 (Dandandan)
  • Eliminate redundant ProjectionExecs #21333 (Dandandan)
  • Minor: add tests for regexp_replace and capture groups #21413 (alamb)
  • bench: add benchmarks for first_value, last_value #21409 (theirix)
  • chore(deps): bump the all-other-cargo-deps group with 4 updates #21435 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.70.3 to 2.74.0 #21434 (dependabot[bot])
  • test: Add datafusion.format.* configs test coverage #21355 (erenavsarogullari)
  • Estimate aggregate output rows using existing NDV statistics #20926 (buraksenn)
  • Follow-up: remove interleave panic recovery after Arrow 58.1.0 #21436 (xudong963)
  • writing table to parquet followed by read and schema check #21444 (Rich-T-kid)
  • chore(deps): bump cryptography from 46.0.6 to 46.0.7 #21489 (dependabot[bot])
  • Preserve logical cast field semantics during physical lowering with field-aware CastExpr #20836 (kosiew)
  • Add more regexp_replace test coverage #21485 (alamb)
  • Introduce Morselizer API, rewrite ParquetOpener to ParquetMorselizer #21327 (alamb)
  • chore: create benches small ints for count_distinct #21521 (coderfender)
  • refactor: extract sort pushdown logic from FileScanConfig into separate module #21457 (zhuqi-lucas)
  • chore: Add array_slice tests for overlapping nulls across inputs #21540 (neilconway)
  • Migrate PhysicalExprAdapter to unified CastExpr and remove CastColumnExpr usage #21493 (kosiew)
  • Unify cast handling by removing CastColumnExpr branches in pruning and ordering equivalence #21545 (kosiew)
  • [datafusion-spark] Add Spark-compatible ceil function #20593 (shivbhatia10)
  • sql: render PostgreSQL array literals as ARRAY[...] in unparser #21513 (xiedeyantu)
  • physical_optimizer: preserve_file_partitions when num file groups < target_partitions #21533 (jayshrivastava)
  • EliminateOuterJoin with Like, IsTrue, IsFalse, IsNotUnknown #21549 (SubhamSinghal)
  • chore(deps): bump hashbrown from 0.16.1 to 0.17.0 #21611 (dependabot[bot])
  • chore(deps): bump ctor from 0.8.0 to 0.10.0 #21612 (dependabot[bot])
  • Rewrite FileStream in terms of Morsel API #21342 (alamb)
  • Consolidate special case regexp_match logic #21486 (alamb)
  • chore(deps): bump taiki-e/install-action from 2.74.0 to 2.75.10 #21605 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 3 updates #21610 (dependabot[bot])
  • bench: first_last remove noisy benchmarks, add update_batch #21487 (theirix)
  • chore: Fix typo problems #21495 (erenavsarogullari)
  • chore(deps-dev): bump follow-redirects from 1.15.6 to 1.16.0 in /datafusion/wasmtest/datafusion-wasm-app #21601 (dependabot[bot])
  • bench: Scale sort benchmarks to 1M rows to exercise merge path #21630 (mbutrovich)
  • Port filter_pushdown.rs async tests to sqllogictest #21620 (adriangb)
  • chore: fix cargo audit and dependencies check on main #21655 (alamb)
  • Spark make_valid_utf8 function implementation #20633 (kazantsev-maksim)
  • chore(deps): update tokio from 1.51 to 1.52 #21670 (ahmed-mez)
  • Use ListArray nullability instead of offsets for array_element, array_any_value. #21672 (tabac)
  • chore: breakdown array.slt into smaller files #21658 (comphead)
  • chore: Add more tests with GROUP BY to test spark collect_set #21659 (comphead)
  • Add strategy-focused InList benchmarks #21648 (geoffreyclaude)
  • Fix massive spill files for StringView/BinaryView columns II #21633 (adriangb)
  • chore: Backport 53.1.0 changelog #21686 (comphead)
  • refactor: Introduce SpillState enum for memory-limited NLJ execution #21636 (viirya)
  • Support Date32/Date64 in unwrap_cast optimization #21665 (Dandandan)
  • feat[expr-common]: add REE arithmetic coercion for numeric and decimal #21179 (asubiotto)
  • Make test_display_pg_json pass regardless of build setup and dependencies #21502 (AdamGS)
  • refactor: Share left-side spill file across partitions on OOM fallback #21699 (viirya)
  • Spark is_valid_utf8 function implementation #21627 (kazantsev-maksim)
  • chore: use bench array helpers from Arrow bench_util #21544 (theirix)
  • chore: add count distinct group benchmarks #21575 (coderfender)
  • minor: More comments to read_spill_as_stream #21713 (2010YOUY01)
  • Dynamic work scheduling in FileStream #21351 (alamb)
  • chore: Update Release instructions #21705 (comphead)
  • test: add tests for spill file sizes to verify View GC #21750 (RatulDawar)
  • chore(deps): bump astral-sh/setup-uv from 8.0.0 to 8.1.0 #21759 (dependabot[bot])
  • chore(deps): bump aws-config from 1.8.15 to 1.8.16 in the all-other-cargo-deps group #21760 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.35.1 to 4.35.2 #21758 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.75.10 to 2.75.18 #21757 (dependabot[bot])
  • Snowflake Unparser dialect and UNNEST support #21593 (yonatan-sevenai)
  • Skip files outside partition structure in hive-partitioned listing tables #21756 (zhuqi-lucas)
  • Handle canceled partitioned hash join dynamic filters lazily #21666 (adriangb)
  • Improve ergonomics for ExecutionPlanMetricsSet and MetricsSet #21762 (gabotechs)
  • [Minor]: unify ANY/ALL planning and align ANY NULL semantics with PG #21743 (buraksenn)
  • [Minor]: fix security audit because of rustls-webpki version #21785 (buraksenn)
  • refactor: Simplify NLJ re-scans with ReplayableStreamSource #21742 (2010YOUY01)
  • ci: permit stale workflow to delete cache #21772 (Jefffrey)
  • Unparser drops ORDER BY alias when flattening Projection through SubqueryAlias #21491 (yonatan-sevenai)
  • chore: re-enable add_months overflow test #21774 (Jefffrey)
  • chore: add aggregation test for listview types #21776 (Jefffrey)
  • chore: re-enable array_union nested null array edge case test #21773 (Jefffrey)
  • Fix: allow coercion from Binary and LargeBinary into BinaryView #21800 (bert-beyondloops)
  • chore: leave specialised bench helpers #21810 (theirix)
  • Add quote style and trimming to csv writier #20813 (xanderbailey)
  • chore(deps): bump picomatch from 2.3.1 to 2.3.2 in /datafusion/wasmtest/datafusion-wasm-app #21164 (dependabot[bot])
  • perf(substr_index): speed up scalar and Utf8View #21754 (kumarUjjawal)
  • Fix PushdownSort dropping LIMIT when eliminating SortExec #21744 (sgrebnov)
  • chore: use Arc::unwrap_or_clone in more places #21823 (Dandandan)
  • build: explicitly set publish = false for internal crates #21869 (rluvaton)
  • chore: bump API limit for stale workflow #21867 (Jefffrey)
  • chore: bump sha & md-5 to 0.11.0 #21840 (Jefffrey)
  • feat : ABI upgrade from abi_stabby to stabby since abi_stable is no longer maintained #21030 (coderfender)
  • Add protobuf serialization/deserialization support for EmptyTable scans #20844 (OlegWock)
  • Support Dictionary Arrays in MIN/MAX Aggregates #21315 (kosiew)
  • Fix some GH action permission issues identified by CodeQL #21838 (Jefffrey)
  • Add support for nested types to nullif. #21764 (tabac)
  • chore(deps): bump taiki-e/install-action from 2.75.18 to 2.75.23 #21887 (dependabot[bot])
  • chore(deps): bump libloading from 0.8.9 to 0.9.0 #21890 (dependabot[bot])
  • refactor array_remove benchmarks & add nested benches #21834 (Jefffrey)
  • Update astral-tokio-tar to appease cargo_audit #21902 (alamb)
  • Remove unnecessary Mutex in SharedMemoryReservation #21899 (gabotechs)
  • ci: add breaking change detector #21499 (rluvaton)
  • Fix GH action permissions in rust.yml and docs.yaml workflows #21884 (Jefffrey)
  • chore: fix iff typos #21904 (comphead)
  • Deduplicate InList primitive static filters #21932 (geoffreyclaude)
  • Fix nesting of permissions block in docs workflow #21930 (Jefffrey)
  • dependencies check are now required to merge ci #21940 (blaginin)
  • build: allow posting comments on PRs made from forks and fix missing protobuf #21913 (rluvaton)
  • Use shared statistics merge for union stats #21430 (kumarUjjawal)
  • Add ClickBench URL pushdown benchmark #21945 (xudong963)
  • test(sqllogictest): stabilize parquet output_rows_skew with WITH ORDER #21898 (RatulDawar)
  • Skip unnecessary plan rebuild in adjust_input_keys_ordering for non-join plans #21947 (zhuqi-lucas)
  • Adding Use of arrow's has_true() / has_false() #21806 (raushanprabhakar1)
  • feat[expr-common]: support regex and LIKE coercion on REE and Dict value types that require an extra coercion step #21924 (asubiotto)
  • feat[expr-common]: support REE in coalesce #21919 (asubiotto)
  • proto: serialize and dedupe dynamic filters v2 #21807 (jayshrivastava)
  • chore: fix datafusion-spark substring #21963 (comphead)
  • Respect DATA_DIR location for sql benchmarks #21961 (Omega359)
  • ci: use base repository branch for breaking change detector #22006 (rluvaton)
  • bench: add to_char_array_date32 #22007 (huymq1710)
  • ci: add auto detected api change label on breaking change detecting in the CI #21953 (rluvaton)
  • Fix fully matched row groups with null counts #21907 (xudong963)
  • functions: Add dict support for get field #21115 (brancz)
  • fix(physical-plan): set column byte_size to 0 in FilterExec zero-row interval stats #21999 (buraksenn)
  • Explicitly declare spill codec dependency in physical-plan #21917 (kosiew)
  • Add benchmark_runner for sql_benchmarks with help and list commands #22001 (Omega359)
  • chore: datafusion-spark substring to support Binary types #21979 (comphead)
  • Add reusable plan-time schema alignment helper and apply to RecursiveQueryExec #21912 (kosiew)
  • Upgrade to arrow-rs / parquet / avro 58.2.0 #21812 (alamb)
  • kill linux-build-lib from extended tests #21227 (blaginin)
  • chore: Rust checks are required + merge queue #21941 (blaginin)
  • Add wide-schema benchmark suite for measuring per-file metadata overhead #21970 (adriangb)
  • chore(deps): bump ctor from 0.10.1 to 1.0.1 #22023 (dependabot[bot])
  • ci: narrow macOS test scope to datafusion-ffi, run benchmarks on amd64 #22048 (blaginin)
  • chore(deps): bump github/codeql-action from 4.35.2 to 4.35.3 #22019 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.74.0 to 2.77.0 #22018 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 2 updates #22022 (dependabot[bot])
  • Allow benchmark allocator features together #21905 (xudong963)
  • Rich t kid/introduce dict benchmarks #21860 (Rich-T-kid)
  • Add benchmarks for dictionary path of new_group_values #22004 (Rich-T-kid)
  • Support IS (NOT) DISTINCT FROM in Unparser #22054 (cetra3)
  • chore: Fix broken build with --benches --all-features #22081 (neilconway)
  • Chore: Fix TPC-DS schema/query (fixes q30 run) #22086 (Dandandan)
  • chore(deps): (fix CI) bump taiki-e/install-action from 2.77.0 to 2.77.6 #22110 (gstvg)
  • Prevent empty grouping sets from being eliminated on empty input #22039 (xiedeyantu)
  • Consolidate and document SQL AST shims #22094 (alamb)
  • Support distinct-from predicates in Parquet pruning #22084 (Dandandan)
  • minor: Track Parquet rows and pages matched when the page index is skipped #22085 (nuno-faria)
  • Update to arrow / parquet from 58.2.0 --> 58.3.0 #22066 (alamb)
  • Add sqllogictest coverage for unused UNNEST pruning edge cases #22074 (kosiew)
  • chore(deps): bump actions/labeler from 6.0.1 to 6.1.0 #22124 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group with 5 updates #22128 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.35.3 to 4.35.4 #22122 (dependabot[bot])
  • mem: Cleanup resources of done streams immediately #22064 (EmilyMatt)
  • Propagate field metadata through NTH_VALUE, FIRST_VALUE, and LAST_VALUE window functions #22112 (paleolimbot)
  • Minor: Disallow async function in lambdas #22097 (gstvg)
  • chore(deps): bump runs-on/action from 2.1.0 to 2.1.2 #22123 (dependabot[bot])
  • bench: remove stale array_expression benchmark #22143 (kumarUjjawal)
  • Add resolve_lambda_variables helper to Expr and LogicalPlan #22101 (gstvg)
  • Fix panic on deep compound identifiers #22186 (Dandandan)
  • Refactor scalar min/max dispatch into function-based helpers #22062 (kosiew)
  • fix missing window expressions when unparsing plans without outer projections #21801 (nathanb9)
  • chore(deps): bump urllib3 from 2.6.3 to 2.7.0 #22109 (dependabot[bot])
  • Call take arrays once per repartitioned input batch #22159 (gene-bordegaray)
  • Refactor parquet row filter setup #22191 (xudong963)
  • fix date_bin overflows subtracting extreme nanosecond timestamp origin #22251 (xiedeyantu)
  • fix date_trunc overflows converting extreme non-ns timestamps to nanoseconds #22262 (xiedeyantu)
  • Extract parquet push decoder module #22289 (xudong963)
  • Track spill read-back memory in SMJ #22103 (SubhamSinghal)
  • refactor(parquet-datasource): split opener.rs into an opener/ module #22346 (adriangb)
  • refactor(parquet-datasource): split sink and schema_coercion out of file_format.rs #22347 (adriangb)
  • fixing negative power to zero #22277 (raushanprabhakar1)
  • refactor(parquet-datasource): split bloom_filter out of row_group_filter.rs #22348 (adriangb)
  • Revert "[Minor]: unify ANY/ALL planning and align ANY NULL semantics with PG (#21743)" #22345 (alamb)
  • Fix pruning predicate for LIKE expressions with escape sequences #22375 (masonh22)
  • Fix: lead/lag extreme offsets handling #22243 (Dandandan)
  • chore(deps): fix CI, bump astral-tokio-tar #22382 (gstvg)
  • chore: Replace stray old-style string builder in substr #22183 (neilconway)
  • chore(deps): bump taiki-e/install-action from 2.77.6 to 2.79.2 #22377 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.35.4 to 4.35.5 #22376 (dependabot[bot])
  • chore(deps-dev): bump webpack-dev-server from 5.2.1 to 5.2.4 in /datafusion/wasmtest/datafusion-wasm-app #22349 (dependabot[bot])
  • chore(deps): bump idna from 3.11 to 3.15 #22381 (dependabot[bot])
  • Refactor Spark format_string numeric %c conversion dispatch #22166 (kosiew)
  • chore(deps): bump sysinfo from 0.38.4 to 0.39.2 #22380 (dependabot[bot])
  • fix regexp_count should count empty-pattern matches #22311 (xiedeyantu)
  • Actually preserve predicate execution order in PushDownFilter #21643 (joroKr21)
  • chore(deps): bump qs and body-parser in /datafusion/wasmtest/datafusion-wasm-app #22321 (dependabot[bot])
  • [branch-54] add changelog #22402 (mbutrovich)
  • [branch-54]: Backport 22404. Fix Spark slice function on negative OOB #22443 (comphead)
  • [branch-54] Cherry-pick #22493: restore SortExec elimination after stats-based file reorder #22501 (zhuqi-lucas)
  • [branch-54] Fix: compact view buffers in ScalarValue::compact for all container types (#21934) #22446 (alamb)
  • [branch-54] Support transparent ExecutionPlan downcasts #22565 (geoffreyclaude)
  • [branch-54] Fix TopK DISTINCT aggregation preserving NULLs (#22571) #22634 (alamb)
  • [branch-54] refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct (#22593) #22635 (alamb)
  • [branch-54] chore: Cleanup and refactor build_join in ScalarSubqueryToJoin (#… #22693 (LiaCastaneda)
  • [branch-54] fix: clear handled OFFSET before child recursion in LimitPushdown (#22525) #22631 (alamb)
  • [branch-54] refactor: give parquet CDC options an explicit enabled flag (backport #22632) #22648 (kszucs)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    70	Neil Conway
    68	dependabot[bot]
    56	Andrew Lamb
    26	Burak Şen
    26	Oleks V
    21	Daniël Heres
    20	Adrian Garcia Badaracco
    18	Kumar Ujjawal
    17	Matt Butrovich
    16	kosiew
    15	Qi Zhu
    15	Tim Saucer
    14	Zhen Chen
    13	Dmitrii Blaginin
    13	Jeffrey Vo
    13	Yongting You
    13	xudong.w
    12	Huaijin
    11	Bhargava Vadlamani
    10	Eren Avsarogullari
    10	Matthew Kim
     9	gstvg
     8	Raz Luvaton
     8	Subham Singhal
     8	theirix
     6	Adam Gutglick
     6	Gabriel
     6	Jonathan Chen
     5	Alessandro Solimando
     5	Alfonso Subiotto Marqués
     5	Bruce Ritchie
     5	Liang-Chi Hsieh
     5	Lía Adriana
     5	Nuno Faria
     5	Sergei Grebnov
     4	Andy Grove
     4	Ariel Miculas-Trif
     4	Geoffrey Claude
     4	Jayant Shrivastava
     4	Kevin Liu
     4	lyne
     3	Brent Gardner
     3	David López
     3	Dewey Dunnington
     3	Harrison Crosse
     3	Huy Mac
     3	Kazantsev Maksim
     3	Konstantin Tarasov
     3	Namgung Chan
     3	Peter L
     3	RIchard Baah
     3	Ratul Dawar
     3	Raushan Prabhakar
     3	Xander
     3	Yonatan Striem Amit
     3	Yu-Chuan Hung
     3	crm26
     2	Acfboy
     2	Adam Curtis
     2	Albert Skalt
     2	Anastasios Bakogiannis
     2	Bert Vermeiren
     2	Frederic Branczyk
     2	Geethapranay1
     2	Jonah Gao
     2	Krisztián Szűcs
     2	Liam Feehery
     2	Marko Grujic
     2	Michael Kleen
     2	Peter Nguyen
     2	Rafael Herrero
     2	Rohan Krishnaswamy
     2	Samyak Sarnayak
     2	Sergey Zhukov
     2	Shiv Bhatia
     2	Tobias Schwarzinger
     2	hsiang-c
     2	jj.lee
     2	linfeng
     2	yaommen
     1	Adam Reeve
     1	Ahmed Mezghani
     1	Alex Zhang
     1	Alexander Alexandrov
     1	Alexander Rafferty
     1	Alexandre Crayssac
     1	Andrey Koshchiy
     1	Asish Kumar
     1	Ben Bellick
     1	Bruno Volpato
     1	Daniel Tu
     1	Druva
     1	EeshanBembi
     1	Emil Ernerfeldt
     1	Emily Matheys
     1	Ernest Provo
     1	Filip Petkovski
     1	Filip Wojciechowski
     1	Florian Müller
     1	Fred Thomas
     1	Gene Bordegaray
     1	Georgi Krastev
     1	Guillaume Boucher
     1	Haresh Khanna
     1	Helgi Kristvin Sigurbjarnarson
     1	Heran Lin
     1	Jamal Saad
     1	Jarro van Ginkel
     1	Jax Liu
     1	Joan Antoni RE
     1	Justin O'Dwyer
     1	Kartik Gupta
     1	Krishna Sudarshan J
     1	Kristin Cowalcijk
     1	Lavkesh Lahngir
     1	Martin Hilton
     1	Mason
     1	Nathan
     1	Oleh
     1	Ramakrishna Chilaka
     1	Rizky Mirzaviandy Priambodo
     1	RyanStewart
     1	Shivaang
     1	Soham Bhattacharjee
     1	Stu Hood
     1	Thomas Tanon
     1	Tim-53
     1	UBarney
     1	Viktor Yershov
     1	Vinay Mehta
     1	Zhang Xiaofeng
     1	aditya singh rathore
     1	alexanderbianchi
     1	blaginin
     1	dario curreri
     1	dd-david-levin
     1	gabriel
     1	nathan
     1	niebayes

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.