ContextQMD
Back to Datafusion

Apache DataFusion 55.0.0 Changelog

dev/changelog/55.0.0.md

55.0.0118.3 KB
Original Source
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

Apache DataFusion 55.0.0 Changelog

This release consists of 877 commits from 175 contributors. See credits at the end of this changelog for more information.

See the upgrade guide for information on how to upgrade from previous versions.

Breaking changes:

  • fix: preserve null_aware on logical JoinNode proto round-trip #22104 (mithuncy)
  • PushdownFilter optimizations #21668 (joroKr21)
  • proto: add proto converter reference to PhysicalExtensionCodec trait #21055 (jayshrivastava)
  • fix ^ evaluates as bitwise XOR instead of exponentiation #22314 (xiedeyantu)
  • Add EnsureRequirements: merged EnforceDistribution + EnforceSorting with idempotent pushdown_sorts #21976 (zhuqi-lucas)
  • feat(physical-expr): DynamicFilterTracker for cheap dynamic-filter change detection #22460 (adriangb)
  • Add minimal APIs / hooks for granular statistics collection in TableProvider implementations #22300 (adriangb)
  • Add lambda substrait support #21193 (gstvg)
  • minor: add Any to QueryPlanner trait #22241 (milenkovicm)
  • Add Physical Partitioning::Range enum variant #22207 (gene-bordegaray)
  • refactor: cache schema_without_virtual_columns and remove TableSchema::with_virtual_columns #22600 (mbutrovich)
  • feat(sql): Postgres-style EXPLAIN (...) option list #21768 (adriangb)
  • refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct #22593 (LiaCastaneda)
  • feat: add pgjson format support for EXPLAIN ANALYZE #21767 (adriangb)
  • Gate new ScalarSubqueryExec node behind session property #22530 (LiaCastaneda)
  • fix: Correctly compute nullability in recursive CTE schemas #22552 (neilconway)
  • Allow specifying an arrow schema for PartitionedFile #22360 (fpetkovski)
  • refactor: give parquet CDC options an explicit enabled flag #22632 (kszucs)
  • Add optimize_with_context to FFI_PhysicalOptimizerRule #22584 (nathanb9)
  • perf(logical-plan): box CreateExternalTable / CreateFunction in DdlStatement (-45% LogicalPlan size) #22733 (zhuqi-lucas)
  • feat: add max_row_group_bytes option to ParquetOptions #22649 (Satyr09)
  • feat: Add Spark SQL parser dialect config #22529 (kumarUjjawal)
  • refactor: Split hash aggregation logic into separated streams #22729 (2010YOUY01)
  • Add logical range partitioning representation #22777 (gene-bordegaray)
  • refactor: centralize SQL dialect metadata #22840 (kumarUjjawal)
  • Revert custom allocator auditing of MemoryPool tracking in SLTs #22860 (avantgardnerio)
  • fix: Correct output-count stats for partitioned partial aggs #22780 (neilconway)
  • fix: preserve async UDF return field metadata #22663 (Kontinuation)
  • fix: preserve Spark next_day whitespace validation #22720 (xfocus3)
  • FFI: plumb placement for FFI_ScalarUDF #22608 (Amogh-2404)
  • refactor: remove opt_filter in GroupsAccumulator::merge_batch #22816 (haohuaijin)
  • feat: decimal support for gcd and lcm #22655 (theirix)
  • refactor: Update SortMergeJoin to use async spill abstractions #22230 (pantShrey)
  • Add MERGE INTO types to datafusion-expr #20763 (wirybeaver)
  • Remove redundant collect_stat and target_partitions on ListingOptions #22969 (gabotechs)
  • fix: Omit NULL values from build side of hash joins #22893 (neilconway)
  • refactor: Simplify approx_distinct (-200 LoC) #22921 (2010YOUY01)
  • Introduce generic memory-limiting cache for parquet metadata #22613 (mkleen)
  • Add StatisticsContext parameter to partition_statistics #21815 (asolimando)
  • feat(parquet): intra-file early stopping via statistics + dynamic filters #22450 (zhuqi-lucas)
  • feat: logical plan protobuf representation for range repartitioning #23030 (saadtajwar)
  • perf: optimize object store requests when reading CSV #22962 (saadtajwar)
  • Group scan time expression rewrite functionality for UDFs in new module in datafusion-physical-expr-adapter #23125 (AdamGS)
  • [physical-plan]: remove deprecated UnionExec::new #23100 (mgkz0)
  • chore(datasource): remove deprecated create_writer free function (Closes #23080 — partial) #23129 (Dodothereal)
  • chore(catalog): remove deprecated ViewTable try_new (Closes #23080 - partial) #23131 (Dodothereal)
  • Add ListingOptions::output_partitioning and FileScanConfig::output_partitioning for pre-defined file partitioning #22657 (gene-bordegaray)
  • chore(parquet): remove deprecated schema-coercion helpers (Closes #23080 - partial) #23132 (Dodothereal)
  • chore(expr): remove deprecated Filter::try_new_with_having (Closes #23080 - partial) #23150 (Dodothereal)
  • chore(common): remove deprecated DFSchema::check_arrow_schema_type_compatible (Closes #23080 - partial) #23151 (Dodothereal)
  • chore(catalog-listing): remove deprecated split_files free fn (Closes #23080) #23152 (Dodothereal)
  • chore(sql): remove deprecated DFParser constructors (Closes #23080 - partial) #23142 (Dodothereal)
  • chore(common): remove deprecated DFSchema type-check method (Closes #23080 - partial) #23144 (Dodothereal)
  • chore(expr): remove deprecated Filter::try_new_with_having (Closes #23080 - partial) #23145 (Dodothereal)
  • chore(expr-common): remove deprecated Signature::get_possible_types (Closes #23080 - partial) #23147 (Dodothereal)
  • [sql]: remove old deprecated DFParser::new and DFParser::new_with_dialect #23101 (mgkz0)
  • chore(common): remove deprecated equivalent_names_and_types (Closes #23080) #23153 (Dodothereal)
  • chore(expr-common): remove deprecated Signature get_possible_types (Closes #23080 - partial) #23135 (Dodothereal)
  • [execution] Remove deprecated disk manager configuration API #23139 (mgkz0)
  • [physical-plan]: remove deprecated spill_record_batch_by_size #23029 (alamb)
  • fix: Fix peak memory display in EXPLAIN ANALYZE for multiple operators #23140 (2010YOUY01)
  • chore(datasource): remove deprecated add_row_stats (Closes #23080 - partial) #23134 (Dodothereal)
  • feat: introduce pluggable SpillFile trait and TempFileFactory for custom spill backends #21882 (pantShrey)
  • Add Distribution::HashPartitioned to Distribution::KeyPartitioned API bridge #23259 (gene-bordegaray)
  • feat: add datafusion.execution.enable_file_stream_work_stealing config #23294 (andygrove)
  • refactor: make file-statistics cache keys schema-aware #23201 (Phoenix500526)
  • perf: preserve dictionary encoding for lower/upper to avoid materializing low-cardinality columns #22905 (lyne7-sc)
  • Remove unstable public methods for DynamicFilterPhysicalExpr after proto migration #23423 (jayshrivastava)
  • refactor: remove redundant partitioned_by_file_group file scan field #23189 (Phoenix500526)
  • Add protobuf support for lambdas #22362 (gstvg)
  • Support co-partitioned range inner equi joins #23184 (gene-bordegaray)
  • refactor: Migrate ScalarSubqueryExpr to self-serialization proto pattern #23130 (mattp5657)
  • refactor(physical-plan): externalize statistics traversal into StatisticsContext #23051 (asolimando)
  • perf: Extend WindowTopN to support RANK #22885 (SubhamSinghal)
  • refactor: make join projection pushdown schema-aware via ColumnIndex/… #23185 (Phoenix500526)
  • ci: reintroduce code coverage reporting with cargo-llvm-cov #23336 (buraksenn)
  • fix: align dictionary coercion across typed signatures #23549 (lyne7-sc)
  • fix: preserve EmptyExec and PlaceholderRowExec partition count across proto round-trip #23643 (andygrove)
  • Resolve lost wakeup in SpillPoolReader with multiple concurrent SpillPoolWriters #23522 (pepijnve)
  • chore: deprecate record_batch macro in favor of upstream one #23295 (buraksenn)
  • fix: time ± interval returns a wrapped time instead of an interval #23279 (vismaytiwari)
  • Add ExecutionPlan try_to_proto / try_from_proto hooks + ProjectionExec reference #23495 (adriangb)
  • feat: Support multiple external table locations #22695 (kumarUjjawal)
  • refactor: pass PhysicalPlanningContext explicitly through planner traits #23649 (timsaucer)
  • feat(proto): thread expr encode/decode context into try_encode_expr / try_decode_expr #23733 (adriangb)
  • refactor(proto): migrate FilterExec serde #23708 (Phoenix500526)
  • refactor(proto): migrate single-child plans #23710 (Phoenix500526)
  • refactor(proto): migrate sort merge join serde #23712 (Phoenix500526)
  • feat: Range Partitioning FFI #23520 (saadtajwar)
  • Bump MSRV from 1.88.0 to 1.94.0 #23632 (Jefffrey)
  • refactor: move catalog traits to session crate #23703 (timsaucer)
  • refactor(proto): migrate SortExec and SortPreservingMergeExec serde #23794 (buraksenn)
  • refactor(proto): migrate UnnestExec serde #23739 (Phoenix500526)
  • refactor(proto): migrate GlobalLimitExec and LocalLimitExec serde #23791 (buraksenn)
  • refactor(proto): migrate RepartitionExec serde #23792 (buraksenn)
  • refactor(proto): migrate CrossJoinExec and NestedLoopJoinExec serde #23834 (buraksenn)
  • refactor(proto): migrate UnionExec and InterleaveExec serde #23782 (buraksenn)
  • refactor(proto): migrate symmetric hash join serde #23736 (Phoenix500526)
  • feat: migrate EmptyExec and PlaceholderRowExec to ExecutionPlan proto hooks #23784 (847850277)
  • Remove GroupsAccumulator::supports_convert_to_state and require convert_to_state #23489 (lyne7-sc)
  • chore: Enable unused_async lint, make some functions sync #23679 (neilconway)
  • refactor(proto): migrate HashJoinExec serde #23853 (buraksenn)
  • refactor(proto): migrate AsyncFuncExec to self-serializing proto #23825 (mattp5657)
  • refactor(proto): migrate window serde #23780 (Phoenix500526)
  • Migrate ExplainExec and AnalyzeExec protobuf serde #23742 (Phoenix500526)
  • refactor(proto): migrate aggregate exec serde #23779 (Phoenix500526)
  • refactor(proto): remove legacy scan field #23445 (Phoenix500526)
  • ScalarUdfImpl::strictly_order_preserving: Allow expression to report whether they keep the same ordering of the input #23807 (rluvaton)
  • FFI: forward ScalarUDF preserves_lex_ordering #23069 (Amogh-2404)
  • perf(functions-aggregate): optimize sliding window MIN/MAX using monotonic deques (#23826) #23827 (pavan51)
  • chore(deps): bump syn from 2.0.119 to 3.0.2 #23945 (dependabot[bot])
  • refactor(proto): migrate scalar subquery serde #23915 (Phoenix500526)
  • refactor: mark the ExecutionPlan proto dispatch traits as non-public API #24001 (adriangb)
  • feat: add GroupColumn support for Duration in multi-column GROUP BY #23783 (tohuya6)
  • refactor: move planning APIs to session crate #23842 (timsaucer)
  • feat: Add support for unnest_outer function for arrays. #22100 (athlcode)
  • perf: track BoundedWindowAggExec Linear-mode watermark once per stream #24033 (neilconway)
  • fix(ffi): preserve aggregate null-handling support #23908 (Amogh-2404)
  • refactor: unify ParquetFileReader and CachedParquetFileReader #24036 (alamb)
  • feat(pruning): expose pruning predicate IN-list rewrite size cap as a config option #24074 (zhuqi-lucas)
  • refactor join-key equality filtering #23843 (shehab-ali)
  • Proto: migrate file sink serialization #23781 (Phoenix500526)
  • refactor(pruning): deprecate PruningPredicate::try_new #24129 (goutamadwant)
  • refactor: move lambda variable scope into Physical Planning Context #23989 (sweb)
  • feat: Implement FFI_QueryPlanner #24028 (timsaucer)
  • add ExecutionPlan::dynamic_expressions_produced() method #24068 (jayshrivastava)
  • fix(proto): preserve HashJoinExec fetch across serialization #24165 (adriangb)
  • refactor(proto): migrate CsvSource serde #24177 (buraksenn)
  • refactor(proto): migrate ParquetSource serde #24169 (buraksenn)
  • refactor(proto): migrate JsonSource serde #24178 (buraksenn)
  • Proto: migrate MemorySourceConfig to per-source try_to_proto / try_from_proto hooks #24187 (adriangb)
  • refactor(proto): migrate AvroSource serde #24190 (buraksenn)
  • refactor(proto): migrate ArrowSource serde #24189 (buraksenn)
  • chore(proto): deprecate AsyncFuncExec::async_exprs, which only existed for proto serialization #24168 (adriangb)
  • Reapply "Add ExecutionPlan::apply_expressions() (apache#20337)" (apache#22437) #24018 (jayshrivastava)
  • fix(proto): serialize Global/LocalLimitExec required_ordering #24183 (buraksenn)
  • fix(proto): preserve AggregateExec schema and reversed state #24207 (buraksenn)
  • perf: remove per-row String allocations from the Spark url functions #23884 (andygrove)
  • Expose accumulator state to allow prefix scanning #24035 (avantgardnerio)
  • fix(lambda): only push referenced params into the merged batch #24162 (LiaCastaneda)
  • Enable dynamic filters for range-partitioned joins #23854 (peterxcli)
  • chore(proto): remove never-released deprecated PhysicalPlanNodeExt scaffolding #24269 (adriangb)
  • Restore the From / TryFrom proto conversions dropped since 54.1.0 #24205 (adriangb)
  • FFI: plumb with_updated_config for FFI_ScalarUDF #22797 (Amogh-2404)
  • fix(physical-plan): CTAS panic on wasm32-unknown-unknown #24275 (kentkwu)
  • fix: ensure new_list respects data_type argument #24029 (Ruchirtripathi)

Performance related:

  • Optimize logical optimizer: skip map_subqueries + in-place rewriting #22298 (adriangb)
  • perf: collapse chained projections in a single optimizer pass; reduce memory usage / recursion #22389 (Dandandan)
  • Fix: compact view buffers in ScalarValue::compact for all container t… #21934 (bert-beyondloops)
  • perf: Optimize translate to use new bulk-NULL string builders #22171 (neilconway)
  • perf: Optimize overlay with new string builder #22182 (neilconway)
  • Optimize metric label cloning #22406 (xudong963)
  • perf: optimize array_replace for scalar needle #22387 (lyne7-sc)
  • perf: optimize array_remove for scalar needle #22390 (lyne7-sc)
  • perf: Optimize split_part using bulk-NULL string builders #22283 (neilconway)
  • perf: hoist split_vec_min_alloc to datafusion-common and shrink the emitted prefix #22416 (RyanJamesStewart)
  • perf(physical-optimizer): skip ensure_distribution rebuild when children are unchanged #22521 (zhuqi-lucas)
  • perf: Handle intermediate Projection nodes in EliminateOuterJoin #22534 (neilconway)
  • perf: array-free fast paths for ScalarValue::cast_to #22576 (alamb)
  • perf(optimizer): EliminateCrossJoin fast-path for join-free plans #22612 (zhuqi-lucas)
  • perf: optimize date subtraction to avoid intermediate array allocation #22591 (lyne7-sc)
  • perf: optimize arrays_zip perfect list zips #22285 (puneetdixit200)
  • perf: Reorder predicates in conjuncts via simple heuristic #22343 (neilconway)
  • perf: avoid unnecessary large allocations #22558 (ariel-miculas)
  • perf: Optimize semi-, anti-join index alignment #22794 (neilconway)
  • perf: improve approx_distinct performance 100x when there are fewer distinct values with many groups #22768 (haohuaijin)
  • perf: fast-path inline strings in ByteViewGroupValueBuilder::vectorized_append #21794 (EeshanBembi)
  • perf: Convert inner joins to semi joins when equivalent #22652 (neilconway)
  • refactor: use raw view access in do_append_val_inner and consolidate duplicated logic #22907 (EeshanBembi)
  • perf: avoid possibly expensive string formatting if no error is encountered #23157 (tschwarzinger)
  • Perf: cache primitive sort key in SortPreservingMerge to drop per-comparison bounds checks #23162 (Dandandan)
  • IN LIST: add UInt16 bitmap filter #23012 (geoffreyclaude)
  • perf: coalesce single-column sort runs to cut merge fan-in #23202 (Dandandan)
  • perf: share encoder/reservation across PartitionedTopKExec partition … #23096 (SubhamSinghal)
  • Optimize Int8 and Int16 integer IN filters #23299 (alamb)
  • feat: Implement state conversion for remaining group accumulators #23275 (lyne7-sc)
  • perf: optimize encode in datafusion-functions #23456 (andygrove)
  • perf: optimize ascii in datafusion-functions #23462 (andygrove)
  • perf: optimize nanvl in datafusion-functions #23458 (andygrove)
  • perf: avoid intermediate slice allocation in Spark slice function #23481 (andygrove)
  • perf: optimize make_date in datafusion-functions #23470 (andygrove)
  • perf: speedup date_part isodow by using DayOfWeekMonday1 #23491 (theirix)
  • perf: optimisation for date_part with seconds #23444 (theirix)
  • perf: optimize round expression #23471 (andygrove)
  • perf: optimize string_trim #23541 (andygrove)
  • perf: optimize date_trunc #23542 (andygrove)
  • perf: Optimize array_has() for array needle #23337 (freakyzoidberg)
  • perf: optimize trunc for scalar precision case (10x faster) #23593 (andygrove)
  • perf: optimize upper (6% faster) #23588 (andygrove)
  • perf: preallocate memory in pad #23586 (theirix)
  • perf: optimize replace (2x faster) #23589 (andygrove)
  • perf: optimize regexp_match for literal pattern usage (20% faster) #23547 (andygrove)
  • perf: avoid per-row copy in Spark hex byte encoding #23473 (andygrove)
  • perf: optimize get_field #23537 (andygrove)
  • perf: optimize regexp_instr (40% faster) #23540 (andygrove)
  • perf: don't re-inline CSE'd expensive expressions in projection pushdown #23459 (fordN)
  • perf: optimize left_right in datafusion-functions #23762 (andygrove)
  • perf: preserve dictionary encoding for bit_length, octet_length, and ascii #23743 (lyne7-sc)
  • perf: optimize LEAD/LAG IGNORE NULLS evaluation #23711 (xudong963)
  • feat: add OR pre-selection short-circuit #22979 (kumarUjjawal)
  • perf: optimize find_in_set (up to 24x faster) #23460 (andygrove)
  • refactor: share hex encoding across datafusion-common, functions, and spark #23766 (andygrove)
  • perf: avoid per-row String allocation in Spark bin and char #23881 (andygrove)
  • IN LIST: add branchless filter for small primitive lists #23014 (geoffreyclaude)
  • perf: optimize array_empty udf #23923 (rluvaton)
  • feat(physical-plan): generic Rows-backed GroupColumn keeps mixed schemas on the column-wise path #23523 (zhuqi-lucas)
  • perf: array_agg() performance improvements #23716 (fred1268)
  • perf: Optimize hashing, null-free fast path for percentile_cont, median #23954 (neilconway)
  • perf: null-free fast path for COUNT(DISTINCT) primitive accumulator #23956 (viirya)
  • perf: precompile formats in to_time #23964 (lyne7-sc)
  • perf: Replace SipHash with foldhash in BoundedWindowAggExec #23984 (neilconway)
  • perf: preserve dictionary encoding for character_length, initcap, and reverse #23930 (lyne7-sc)
  • perf: skip re-slicing window partition batches with nothing to prune #24047 (neilconway)
  • perf: gather Linear-mode window input more efficiently #24034 (neilconway)
  • perf: use Vec in ArrowBytesMap #24071 (Punisheroot)
  • perf: preallocate RowsGroupColumn buffers in take_n #24070 (saadtajwar)
  • feat: add GroupColumn support for Decimal256 in multi-column GROUP BY #23849 (tohuya6)
  • perf: preserve dictionary encoding for btrim, ltrim, and rtrim #24100 (lyne7-sc)
  • Skip page index load (and ParquetMetaData clone) when the file has no page index #24150 (alamb)
  • perf: optimize char -> byte offset mapping in regexp_count #24153 (neilconway)
  • perf: skip evaluating fully calculated window partitions #24127 (neilconway)
  • perf: prune window state only for partitions that made progress #24148 (neilconway)

Implemented enhancements:

  • feat: fix slice function on OOB ranges #22404 (comphead)
  • feat: Analyze VALUES for nullability #22089 (neilconway)
  • feat: Add Spark-compatible monthname function to datafusion-spark #21639 (JeelRajodiya)
  • feat: Improve display of Decimal values #22500 (neilconway)
  • feat(catalog): expose InformationSchemataBuilder as public API #22499 (zfarrell)
  • feat: add array_scale scalar function #22466 (crm26)
  • feat: adds array_add function #22459 (SubhamSinghal)
  • feat: add TableSchemaBuilder and store partition columns as Fields #22496 (adriangb)
  • feat: lower repartition_file_min_size default from 10 MiB to 1 MiB #22439 (adriangb)
  • feat: Plumb Parquet virtual columns (row_number) through TableSchema and ParquetOpener #22026 (mbutrovich)
  • feat: add SparkPow UDF returning Infinity for pow(0, negative) #22605 (Brijesh-Thakkar)
  • feat: add array_subtract scalar function #22556 (SubhamSinghal)
  • feat: support Boolean in approx_distinct #22707 (JeelRajodiya)
  • feat: implement retract_batch for array_agg(DISTINCT) sliding window #22719 (SubhamSinghal)
  • feat: add DataFrame fill_nan #22702 (Nagato-Yuzuru)
  • feat: add array_sum scalar function #22542 (crm26)
  • feat: Support IEEE 754 negative zero semantics #22835 (comphead)
  • feat: Add From<Option<T>> trait for Precision enum #22792 (devanbenz)
  • feat: implement Spark-compatible weekday function #22740 (sjhddh)
  • feat(spark): add concat_ws with array support #20928 (davidlghellin)
  • feat: support reading from stdin in datafusion-cli #22839 (huan233usc)
  • feat(unparser): support binary literals #23001 (zyuiop)
  • feat: warn on NULL equality predicates #22948 (ametel01)
  • feat: support file-level parquet row selections #22940 (haohuaijin)
  • feat: support mixed binary and string types for concat UDFs #22244 (theirix)
  • feat(unparser): support DISTINCT FROM operators in the MySQL dialect #22999 (zyuiop)
  • feat: Add new input_file_name UDF for file-backed scans #22978 (AdamGS)
  • feat: add array_avg scalar function #23168 (crm26)
  • feat: Support Decimal type in approx_distinct #23190 (mkleen)
  • feat: Support interval type in approx_distinct #23234 (mkleen)
  • feat: Re-spill sort stream if unable to reserve for 2 streams #22945 (EmilyMatt)
  • feat: Expose cache hits in statistics_cache function #23253 (mkleen)
  • feat: Eagerly drop last finished stream in FusedStreams #23283 (rluvaton)
  • feat: cap spill merge fan-in #23066 (yinli-systems)
  • feat: Support duration type in approx_distinct #23291 (mkleen)
  • feat: Allow datafusion-ffi to opt out of proto parquet #22951 (Xuanwo)
  • feat: Support BinaryView type in approx_distinct #23333 (mkleen)
  • feat: support decimals in trunc UDF #23320 (theirix)
  • feat: add strictness metadata for scalar UDF null propagation and use it in outer join elimination #23148 (lyne7-sc)
  • feat: physical execution for range partitioning #23231 (saadtajwar)
  • feat: Support FixedSizedBinary type for approx_distinct #23417 (mkleen)
  • feat: Support List/ListView types in approx_distinct #23443 (mkleen)
  • feat: add array_first higher-order array function #23267 (EdsonPetry)
  • feat: Expose cache hits in list_files_cache function #23439 (mkleen)
  • feat: Support Map type in approx_distinct #23526 (mkleen)
  • feat: allow Partitioning::Range to satisfy window Distribution::KeyPartitioned requirements #23416 (mithuncy)
  • feat: benchmark_runner, improve --list, optional DATA_DIR #23354 (Omega359)
  • feat: Support Struct type in approx_distinct #23663 (mkleen)
  • feat: allow Full joins to reuse range co-partitioning in HashJoinExec #23583 (mattp5657)
  • feat: support co-partitioned range right-side equi hash joins #23484 (gmhelmold)
  • feat: complete range repartition physical planning #23617 (saadtajwar)
  • feat: Support Union type in approx_distinct #23714 (mkleen)
  • feat: add validating non-Arrow TDigest constructor and accessors #23737 (adriangb)
  • feat: add Spark-compatible hypot function #23774 (KarpagamKarthikeyan)
  • feat: add BuildHasher variants for hash_utils #21820 (xudong963)
  • feat: support ansi for elt #23928 (comphead)
  • feat: centralizing higher-order list lambda evaluation helpers #23911 (saadtajwar)
  • feat: add GroupColumn support for Float16 in multi-column GROUP BY #23785 (tohuya6)
  • feat: switch VirtualTable producer to use expressions field instead of deprecated values #23672 (eliot1480)
  • feat: add GroupColumn support for Interval in multi-column GROUP BY #23786 (tohuya6)
  • feat: drop generator on error to free memory faster #23967 (rluvaton)
  • feat: eliminate LEFT/RIGHT JOINs with redundant sides #23566 (simonvandel)
  • feat(parquet): multi-column lexicographic stats reorder for TopK sort pushdown #23888 (zhuqi-lucas)
  • feat: add Spark-compatible atan2 function #23962 (KarpagamKarthikeyan)
  • feat: Calculate non-distinct sum from column statistics when available #23863 (AdamGS)
  • feat: prune unread Parquet leaves when a nested column is cast to a narrower type #24090 (mbutrovich)
  • feat: Add SQL planner, physical planner, and TableProvider hook for MERGE INTO #22988 (wirybeaver)
  • feat(dataframe): add f16 support to dataframe! macro #24234 (cj-zhukov)

Fixed bugs:

  • fix: indentation for markdown block comments in docstrings #22409 (ariel-miculas)
  • fix(unparser): fold Limit/Sort into outer SELECT when Projection claims Aggregate through them #21375 (yonatan-sevenai)
  • fix(substrait): dedupe names of aggregate measures, not just groupings #22453 (LiaCastaneda)
  • fix: Operator::returns_null_on_null() should include string concat (||) #22458 (neilconway)
  • fix: custom_datasource example ignores projection pushdown in execute() #22417 (kumarUjjawal)
  • fix: handle IS TRUE correctly in EliminateOuterJoin #22444 (neilconway)
  • fix: avoid panic in TableSchema::with_table_partition_cols on shared Arc #22372 (adriangb)
  • fix: avoid panic in date_bin compute_distance near i64::MIN #22408 (SAY-5)
  • fix: make array null argument handling follow SQL semantics #22508 (kumarUjjawal)
  • fix: Set Substrait output types for expressions #20597 (wlhjason)
  • fix: clear handled OFFSET before child recursion in LimitPushdown #22525 (kumarUjjawal)
  • fix: Avoid precision loss for atan2 with integer args #22516 (neilconway)
  • fix: LIKE 'prefix%' pruning fails on Utf8View and LargeUtf8 columns #22562 (lyne7-sc)
  • fix: widen power(decimal, float) to Float64, fix bugs #22482 (neilconway)
  • fix: reborrow metadata values when intersecting union metadata #22491 (officialasishkumar)
  • fix: Correct join cardinality estimation for semi and anti joins with disjoint column ranges #22674 (neilconway)
  • fix: Projection stats Absent for columns referenced >1 time #22679 (neilconway)
  • fix(substrait): plan nested projected window expressions #22630 (bvolpato)
  • fix: render binary columns as hex in DataFrame::describe() #21728 (diegoQuinas)
  • fix: wrong precision in a decimal256 log test #22578 (theirix)
  • fix: Avoid panic decoding invalid parquet writer version from proto #22467 (fallintoplace)
  • fix: make PushDownLeafProjections work with unnest #22620 (pabadrubio)
  • fix: correct cross join byte size statistics #22700 (neilconway)
  • fix: Improve consistency of per-column stats on FilterExec output #22718 (neilconway)
  • fix: Correct computation of selectivity for multi-key joins #22725 (neilconway)
  • fix: replace with empty search string should be a no-op #22497 (Amogh-2404)
  • fix: Remove power(decimal, int) code path #22651 (neilconway)
  • fix: avoid extraneous casts for equivalent nested types #20945 (feichai0017)
  • fix: handle NULLs in sliding SUM(DISTINCT) window frames #22755 (kumarUjjawal)
  • fix: Scale semi/anti-join column stats by estimated row count #22762 (neilconway)
  • fix: preserve timestamp precision when coercing mixed time units #22759 (fengys1996)
  • fix: make skip_partial_aggregation_probe_ratio_threshold match the docs #22752 (haohuaijin)
  • fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions #22791 (nathanb9)
  • fix: Optimize projections in recursive CTEs #22476 (nuno-faria)
  • fix: Coerce aggregate FILTER predicates to boolean #22774 (pchintar)
  • fix: approx_distinct over-counts for utf8view #22815 (haohuaijin)
  • fix: regex simplification of anchored patterns produces wrong results #22727 (lyne7-sc)
  • fix: add backtrace for assert_*_or_internal_err helpers #18910 (rluvaton)
  • fix: map() fails when keys are literals and values are column expressions #22784 (nathanb9)
  • fix: Avoid incorrectly rounding large integers in nanvl #22575 (neilconway)
  • fix: Enable sliding window execution for covar_pop, covar_samp, and corr #22764 (pchintar)
  • fix: handle date_bin negative subsecond and overflow cases #22610 (kumarUjjawal)
  • fix: TRY_CAST returns NULL for timestamp/date overflow #22897 (fengys1996)
  • fix: count shared buffers once in hash join build-side memory accounting #22862 (jordepic)
  • fix(topk): call attempt_early_completion when filter rejects entire batch #22852 (ajegou)
  • fix: Disable join dynamic filters for null-equal joins #22965 (neilconway)
  • fix: ProjectionPushdown internal error on NestedLoopJoin mark joins #22902 (lyne7-sc)
  • fix: parquet limit pruning for row group selections #22942 (haohuaijin)
  • fix: isolate anonymous file statistics cache #22950 (kumarUjjawal)
  • fix: Parquet bloom filter pruning can incorrectly filter decimals encoded as FIXED_LEN_BYTE_ARRAY #22995 (lyne7-sc)
  • fix: Consider column names' case when aliasing tables #22917 (nuno-faria)
  • fix: prevent unparser stack overflow on deeply nested expressions #23058 (adriangb)
  • fix: block timestamp precision narrowing unwrap #22837 (discord9)
  • fix: preserve no-filter SMJ matches across pending outer batches #23049 (neilconway)
  • fix(proto): honor ExecutionPlan downcast_delegate during serialization #23154 (geoffreyclaude)
  • fix: add assert to HashJoinExec::swap_inputs #23078 (haohuaijin)
  • fix: preserve empty projection when ser/de HashJoinExec and NestedLoopJoinExec #23082 (haohuaijin)
  • fix: array_compact handle edge case with NULLs #23192 (comphead)
  • fix(spark): return error from ELT coerce_types when fewer than 2 args #23164 (davidlghellin)
  • fix: Preserve integer values in round() for large Int64 and UInt64 inputs #22697 (pchintar)
  • fix: surface BufferExec input panics instead of silently truncating output #23243 (Tristan1900)
  • fix: apply recursive CTE column-list aliases to the static term #23098 (tomsanbear)
  • fix: unparse columns of stacked pushdown projections unqualified #23176 (Phoenix500526)
  • fix(sort): record output_batches, output_bytes and end_time for when not using merge sort #22878 (rluvaton)
  • fix: Handle decimal columns consistently in SLT tests #23161 (AdamGS)
  • fix: avoid panic parsing non-ASCII runtime config values #23316 (ByteBaker)
  • fix: avoid global SQL stack guard mutation in unparser #23284 (ametel01)
  • fix: Avoid panicing when stats are not available for a file group split #23277 (mkleen)
  • fix: gate debug-only assertions in physical planner test test_optimization_invariant_checker #23323 (buraksenn)
  • fix: Reject out-of-range ArrayMap probe keys on 32-bit targets #22911 (neilconway)
  • fix: return execution error instead of capacity overflow panic in array_resize #23306 (buraksenn)
  • fix: cardinality returns incorrect results for ragged nested arrays #23271 (lyne7-sc)
  • fix: cast [] to FixedSizeList(0, _) #23381 (Jefffrey)
  • fix: don't duplicate volatile expressions when pushing projection into file scan #23395 (fordN)
  • fix: fix typo on doc #23457 (Rich-T-kid)
  • fix: Batch size limit in re-spill compounds #23286 (EmilyMatt)
  • fix: ensure a maximum of buffer_len RecordBatches are cached in spawn_buffered #23560 (ariel-miculas)
  • fix: close the markdown block in docstring #23562 (ariel-miculas)
  • fix: preserve range partitioning through joins #23584 (EdsonPetry)
  • fix: optimize_projections failure with struct-field join keys #22903 (kumarUjjawal)
  • fix: Handle potential overflow in internal state for avg(decimal) #22714 (AdamGS)
  • fix: support type coercion for MAP literals with NULL values in VALUES lists #23521 (PG1204)
  • fix: handle interleaved HashJoin projections in sort pushdown #23591 (xudong963)
  • fix: do not remove DISTINCT when a unique key was downgraded by a join #23548 (simonvandel)
  • fix: preserve aggregate scope when unparsing #23327 (Phoenix500526)
  • fix: keep null-aware anti-join NULLs in the pushed dynamic filter #23104 (mdashti)
  • fix: handle null date and timestamp format arguments #23641 (lyne7-sc)
  • fix: prevent LEAD/LAG IGNORE NULLS panic without null bitmap #23706 (xudong963)
  • fix: Preserve metadata when a cross-join is swapped #23605 (mkleen)
  • fix: coerce SIMILAR TO operands to a common string type #23704 (u70b3)
  • fix: Capture global ORDER BY requirement under ScalarSubqueryExec root #23677 (sgrebnov)
  • fix: avoid overflow in join cardinality estimation #23788 (xudong963)
  • fix: unwrap identity Date cast in comparison unwrapping #23727 (adriangb)
  • fix: reject nested aggregate functions (e.g. sum(sum(x))) during logical planning #23813 (adriangb)
  • fix: array_any_value returns NULL for empty list elements #23775 (bjchambers)
  • fix: fixed decode buffer size estimate for BinaryViewArray #23765 (liningpan)
  • fix: grouped first_value/last_value FILTER excludes NULL predicate rows #23707 (u70b3)
  • fix: NOT IN with NULL subquery returns wrong results under SortMergeJoin #22810 (nathanb9)
  • fix: align physical CASE nullability through casts #23844 (friendlymatthew)
  • fix: Handle null-aware joins correctly in FilterNullJoinKeys when its enabled #23848 (AdamGS)
  • fix: don't infer join predicates for null-aware joins in push_down_filter #23901 (viirya)
  • fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key #23173 (mdashti)
  • fix: Handle input_file_name() pushdown into ParquetSource with filter pushdown enabled #23638 (AdamGS)
  • fix: exclude precision-losing integer-to-float conversions from CastExpr::check_bigger_cast (#23808) #23809 (getChan)
  • fix: eliminate group by constant empty input #22132 (HairstonE)
  • fix: sliding window min() returns wrong value for all-NULL windows #23874 (neilconway)
  • fix: correct percentile_cont(DISTINCT) accumulation and sliding-window retract #23913 (viirya)
  • fix: support parentheses for negative decimal formatting #23718 (wangzhigang1999)
  • fix: last value accumulator merge indexing #23905 (peterxcli)
  • fix: accept LargeUtf8 and Utf8View patterns in SIMILAR TO planning #23735 (u70b3)
  • fix: preserve aggregate filter pushdown order #22926 (discord9)
  • fix(common): preserve an exact zero through filter selectivity estimation #23936 (asolimando)
  • fix: preserve dictionary-value nulls in scalar regex operators #23966 (discord9)
  • fix(datasource): avoid over-conservative transformation of num_rows statistics in file scan config #23670 (tschwarzinger)
  • fix: keep a CoalescePartitionsExec required by a SinglePartition child #23948 (adriangb)
  • fix: TopK aggregation drops groups whose MIN/MAX value is NULL #23684 (u70b3)
  • fix(sql): preserve source qualifiers in CTAS with explicit schema #23879 (lyne7-sc)
  • fix: reject nested arrays in array_distance #23995 (2010YOUY01)
  • fix: Improve error message for metadata conflict in schema #23952 (mkleen)
  • fix: handle empty patterns in regexp_instr #24054 (iamhaseebn)
  • fix: prevent incorrect results when pushing filters through anti joins #24045 (buraksenn)
  • fix: UnionExec now conforms each batch to the union's declared schema #23861 (dariocurr)
  • fix: do not derive ordering for arithmetic that can overflow #23910 (buraksenn)
  • fix: preserve projection field metadata during physical planning #23981 (subotac)
  • fix(proto): preserve empty projection when ser/de MemoryScanExec #24087 (buraksenn)
  • fix: Fix nullability of logical InSubquery expression #23429 (AdamGS)
  • fix: keep every spilled slice of a sort-merge join inner key group #24056 (buraksenn)
  • fix: reduce peak memory usage when round robin tiebreaker is disabled #23606 (ariel-miculas)
  • fix: preserve total_byte_size in calculate_total_byte_size when num_r… #24027 (bert-beyondloops)
  • fix: box aws-config loading future avoid clippy warning #24175 (neilconway)
  • fix: Correctly process numeric literals with underscores #24046 (nuno-faria)
  • fix: typo for the builder error type #24052 (JosephLenton)
  • fix: support untyped NULL input for median #24104 (Sigma-Ma)
  • fix(proto): prevent logical plan serialization stack overflow #24124 (mithuncy)
  • fix: prevent next_day panic on far-future start dates #24194 (viirya)
  • fix: return error instead of panic when decoding ParquetScan/AvroScan without features #24198 (nam2ee)
  • fix: re-enable null-equal join dynamic filters with an IS NULL predicate #23106 (mdashti)
  • fix: generate_series overflow panics at i64 boundary and out-of-range dates #23723 (u70b3)
  • fix: clear stale sliding aggregate state for empty RANGE frames #24185 (lyne7-sc)
  • fix(parquet): remap sorting columns for partitioned writes #24211 (xudong963)
  • fix: reject max_buffered_batches_per_output_file values below 2 #24204 (DevShiba)
  • fix: avoid buffering unbounded repartition output indefinitely #24193 (goutamadwant)
  • fix: infer placeholder types in GROUP BY, HAVING, QUALIFY and ORDER BY (fix for #24042) #24043 (Braedon-Wooding-Displayr)
  • fix: Propagate NULLs in regexp_count, regexp_instr #24239 (neilconway)
  • fix: preserve NULL semantics in log and power simplification #24247 (lyne7-sc)

Documentation updates:

  • Revert "Add ExecutionPlan::apply_expressions() (#20337)" #22437 (alamb)
  • docs: add agent skill for datafusion-ffi crate patterns #22327 (timsaucer)
  • docs: clarify difference between try_cast_literal_to_type and ScalarValue::cast_to #22592 (alamb)
  • added support for MapFromEntries #21720 (athlcode)
  • chore: update Rust toolchain to 1.96.0 #22611 (Dandandan)
  • chore(deps): update pydata-sphinx-theme requirement from <1,>=0.17.1 to >=0.18.0,<1 in /docs #22540 (dependabot[bot])
  • Track allocator-level memory vs MemoryPool during SLTs to prevent OOMs #22626 (avantgardnerio)
  • Add array_product UDF #22703 (SubhamSinghal)
  • docs: revise OptimizerRule trait method descriptions #22582 (jiengup)
  • docs: add Boston DataFusion meetup #22722 (alamb)
  • Add example for PartitionedFile schema #22809 (fpetkovski)
  • [main] Update version and changelog to 54.0.0 #22855 (alamb)
  • docs: link release tracking issue to release management page #22822 (alamb)
  • chore: Define backport criteria #22766 (comphead)
  • docs: Update/improve SELECT reference #22672 (neilconway)
  • docs: link to 2026 Q3-Q4 roadmap discussion #22884 (alamb)
  • refactor(hash-aggr): Migrate the partial aggregation skip optimization to the new hash aggregation impl #22899 (2010YOUY01)
  • Add file_row_index UDF to query file-level row indexes from Parquet files #22604 (AdamGS)
  • chore(deps): update maturin requirement from <2,>=1.13.3 to >=1.14.0,<2 in /docs #22974 (dependabot[bot])
  • chore(deps): update pydata-sphinx-theme requirement from <1,>=0.18.0 to >=0.19.0,<1 in /docs #22972 (dependabot[bot])
  • docs: clarify stdin store buffers on construction, not first use #23060 (huan233usc)
  • Docs: Add PartialSortExec documentation #23092 (alamb)
  • docs: Add Shanghai Apache DataFusion Meetup to events page #23025 (alamb)
  • chore(deps): update maturin requirement from <2,>=1.14.0 to >=1.14.1,<2 in /docs #23117 (dependabot[bot])
  • Add Hotdata to the "known users" list in introduction.md #23004 (zfarrell)
  • doc: More comments on GroupedHashAggregateStream refactor #23200 (2010YOUY01)
  • docs: show struct-returning aggregate window metadata pattern #23248 (ametel01)
  • Align DataFrame::fill_null column argument with fill_nan #22904 (Nagato-Yuzuru)
  • v54 upgrade guide: Remove unreleased-note #23331 (simonvandel)
  • docs: document ClickBench setup details #23315 (ByteBaker)
  • docs: add DataFusion Ballista to related subproject #23377 (coderfender)
  • chore(deps): update setuptools requirement from <83,>=82.0.1 to >=83.0.0,<84 in /docs #23361 (dependabot[bot])
  • [codex] chore: update Rust toolchain to 1.96.1 #23379 (alamb)
  • docs: add infino to known users #23383 (savannahar68)
  • Update Rust toolchain to 1.97.0 #23430 (Dandandan)
  • chore(deps): update pydata-sphinx-theme requirement from <1,>=0.19.0 to >=0.20.0,<1 in /docs #23551 (dependabot[bot])
  • doc: More comments to aggregate planning overview #23525 (2010YOUY01)
  • docs: add partitioned ClickBench SQL example #23637 (ByteBaker)
  • docs: Update committer and PMC list #23621 (alamb)
  • chore: Fix duplicated word typos in comments #23662 (jackylee-ch)
  • Add any_value aggregate function #23043 (yinli-systems)
  • docs: update Polygon.io reference to Massive.com #23734 (xudong963)
  • chore: Update version 54.1.0, add changelog (#23689) #23764 (mbutrovich)
  • docs: add Supermetal to known users #23790 (kumarUjjawal)
  • chore: remove Github filter status:success for pending PR shield #23846 (comphead)
  • Add codecov badge to README #23860 (Jefffrey)
  • docs: add datapress to known users list #23919 (jeroenflvr)
  • test: add regression coverage and docs for NULL format handling #23669 (U0001F3A2)
  • docs: Fixes incorrect type name in UserDefinedLogicalNode comment #23992 (vikrantmehta123)
  • docs: Add more documentation about PartialSortExec operator #24048 (alamb)
  • Docs: Update PR template to ask for user-visible rationale #24053 (alamb)
  • docs: document all fields and methods of DFParquetMetadata #24037 (alamb)
  • Fix syntax examples of some functions #23212 (Viicos)
  • chore(deps): Update to arrow/parquet 59.2.0 #24030 (alamb)
  • Fix duplicated words in documentation #24176 (latent-9)
  • chore: fix some scalar function docs #24134 (Jefffrey)
  • Docs: Add community showcase to the docs page #24217 (alamb)
  • docs: explain Parquet content-defined chunking #24155 (goutamadwant)
  • docs: add IceGate to the list of featured data platforms #24240 (frisbeeman)
  • Docs: Add PR review guide #24051 (alamb)
  • [branch-55] Update additional references to version number #24295 (timsaucer)
  • [branch-55] Backport of refactor(physical-plan): Simplify ExecutionPlan API with replace_children #24296 (JSOD11)

Other:

  • chore: protect branch-53 and branch-54 #22403 (mbutrovich)
  • refactor(parquet-datasource): extract DecoderProjection from build_stream #22398 (adriangb)
  • Fix: Infer placeholder type from subquery #22436 (HairstonE)
  • Split proto serialization to encapsulate private state (#21835) #21929 (adriangb)
  • test: add more tests and docs for heap size estimation #22358 (mkleen)
  • chore: Cleanup and refactor build_join in ScalarSubqueryToJoin #22316 (neilconway)
  • Fix missing field partitioned_by_file_group in serialization #22365 (marc-pydantic)
  • chore: Add existence (semi / anti ) benchmarks for hashjoinexec #21821 (coderfender)
  • chore(deps): bump qs and express in /datafusion/wasmtest/datafusion-wasm-app #22469 (dependabot[bot])
  • chore: Disallow reserve() in clippy to prevent panics #22386 (2010YOUY01)
  • Support DISTINCT ON with aggregation and windows #22169 (kumarUjjawal)
  • Benchmark multi-column GROUP BY performance #22322 (nathanb9)
  • fix(sort-pushdown): restore SortExec elimination after stats-based file reorder #22493 (zhuqi-lucas)
  • fix array_repeat capacity overflow on constant scalar with large count #22305 (xiedeyantu)
  • fix sqrt(-1.0::float8) should error, not return NaN #22308 (xiedeyantu)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 9 updates #22470 (dependabot[bot])
  • Port LikeExpr to use try_to_proto / try_from_proto #22471 (jx2lee)
  • chore(deps-dev): bump fast-uri from 3.1.0 to 3.1.2 in /datafusion/wasmtest/datafusion-wasm-app #22083 (dependabot[bot])
  • refactor: port InListExpr to use try_to_proto/try_from_proto hooks #22503 (kkrainov)
  • refactor(physical-expr): add proto ctx expr helpers and adopt in InList/Like #22513 (adriangb)
  • minor: Make union_with_mix_of_presorted_and_explicitly_resorted_inputs_impl cross platform #22478 (nuno-faria)
  • refactor: add try_to_proto to HashTableLookupExpr #22451 (AnuragRaut08)
  • Simplify get_field over inline struct constructors #22239 (adriangb)
  • Add regression coverage for DATE interval overflow #22519 (puneetdixit200)
  • Make DiskManager max_temp_directory_size dynamically adjustable #22246 (Bukhtawar)
  • chore(deps): bump taiki-e/install-action from 2.79.2 to 2.79.8 #22537 (dependabot[bot])
  • chore(deps): bump actions/stale from 10.2.0 to 10.3.0 #22536 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.35.5 to 4.36.0 #22535 (dependabot[bot])
  • chore(deps): bump log from 0.4.29 to 0.4.30 in the all-other-cargo-deps group #22539 (dependabot[bot])
  • test: add test that validate partial reduce with different number of state fields #21175 (rluvaton)
  • chore: fix two comment typos #22524 (mvanhorn)
  • port NegativeExpr to use the try_to_proto / try_from_proto hooks #22483 (kevinhongzl)
  • chore: update sqllogictest priority list with latest timing summary (8s --> 6s) #22549 (alamb)
  • Support transparent ExecutionPlan downcasts #22559 (geoffreyclaude)
  • Return None for cardinality overflow #22309 (jx2lee)
  • Fix correlated subquery empty defaults for regr_count and approx_distinct #22319 (nathanb9)
  • refactor: port HashExpr proto hooks #22502 (nanookclaw)
  • Port CastExpr to proto hooks #22569 (feichai0017)
  • ci(breaking-change-detector): don't use maintain-one-comment and instead do it manually #22568 (rluvaton)
  • refactor(physical-expr-common): add proto helpers for the recurring shapes in #22418, port already-migrated exprs #22596 (adriangb)
  • Port NotExpr proto hooks #22463 (Herrtian)
  • Migrate UnKnownColumn proto hooks #22464 (koopatroopa787)
  • refactor: Port IsNotNullExpr proto serialization hooks #22532 (chakkk309)
  • Optimize Parquet metadata row-group level statistics collection #22462 (AdamGS)
  • refactor: Port IsNullExpr proto serialization hooks #22509 (chakkk309)
  • chore: Fix typos in comments #22625 (neilconway)
  • Fix TopK DISTINCT aggregation preserving NULLs #22571 (kumarUjjawal)
  • Add range partitioning sqllogictest fixture #22607 (gene-bordegaray)
  • fix(physical-plan): make HashJoinExec dynamic filter pushdown idempotent #22523 (wirybeaver)
  • minor: Improve error message for invalid column expression in SELECT statement #22486 (2010YOUY01)
  • fix(physical-optimizer): make OutputRequirements idempotent #22522 (wirybeaver)
  • fix(array_agg): reverse ordering_values in state() when accumulator is reversed #22597 (ologlogn)
  • chore: Add primary key constraints for TPC-H, TPC-DS #22646 (neilconway)
  • test: cover regexp_like multiline flag #22284 (nanookclaw)
  • test: make push_down_filter_regression dynamic filter content deterministic (#22621) #22643 (diegoQuinas)
  • Revert addition of benchmark_runner for sql_benchmarks #22624 (Omega359)
  • fix array_repeat scalar path overflows total repeated-value count #22274 (xiedeyantu)
  • Refactor Spark format_string integer conversion dispatch #22388 (kosiew)
  • chore: Make sqllogictest pass with default features #22619 (AdamGS)
  • refactor: Port TryCastExpr proto serialization hooks #22550 (chakkk309)
  • fix nth_value window function negates i64::MIN #22304 (xiedeyantu)
  • sqllogictest: account before alloc to avoid panic-after-alloc hazards #22742 (avantgardnerio)
  • chore(deps): bump taiki-e/install-action from 2.79.8 to 2.81.3 #22745 (dependabot[bot])
  • chore(deps): bump github/codeql-action from 4.36.0 to 4.36.1 #22746 (dependabot[bot])
  • chore(deps): bump actions/checkout from 6.0.2 to 6.0.3 #22748 (dependabot[bot])
  • chore(deps): bump astral-sh/setup-uv from 8.1.0 to 8.2.0 #22747 (dependabot[bot])
  • fix date_bin overflows scaling extreme Timestamp(Second) source #22315 (xiedeyantu)
  • test: benchmarks and SLT tests for push-down TopK through join #22760 (adriangb)
  • Refactor hash join build-report lifecycle into BuildReportHandle #22623 (kosiew)
  • Mark BufferExec and AnalyzeExec as eager #22711 (geoffreyclaude)
  • feat(physical-expr): port Literal to try_to_proto / try_from_proto hooks #22636 (koopatroopa787)
  • Add clickbench SQL benchmark #22633 (Omega359)
  • Add imdb SQL benchmark #22680 (Omega359)
  • Add partitioning compatibility API #22590 (gene-bordegaray)
  • Add h2o SQL benchmark #22660 (Omega359)
  • chore(deps): bump the all-other-cargo-deps group with 6 updates #22750 (dependabot[bot])
  • test: make ensure_requirements tests deterministic #22789 (kumarUjjawal)
  • Spark quote function implementation #22642 (kazantsev-maksim)
  • coerce Union vs scalar in comparisons #22825 (friendlymatthew)
  • bench: add predicate_eval SQL micro-benchmark suite for conjunctive filter evaluation #22704 (adriangb)
  • minor: More comments to AggregateMode::PartialReduce #22800 (2010YOUY01)
  • bench: make wide_schema honor DATA_DIR like the other sql_benchmarks #22836 (adriangb)
  • refactor: Port CaseExpr proto serialization hooks #22838 (chakkk309)
  • chore(deps): bump github/codeql-action from 4.36.1 to 4.36.2 #22842 (dependabot[bot])
  • Add tpcds SQL benchmark #22801 (Omega359)
  • chore(deps): bump taiki-e/install-action from 2.81.3 to 2.81.8 #22841 (dependabot[bot])
  • Add hj SQL benchmark #22802 (Omega359)
  • chore(deps): bump the all-other-cargo-deps group with 3 updates #22844 (dependabot[bot])
  • add clickbench sorted SQL benchmark #22807 (Omega359)
  • Add nlj SQL benchmark #22805 (Omega359)
  • Add clickbench extended SQL benchmark #22804 (Omega359)
  • Add smj SQL benchmark #22803 (Omega359)
  • chore(deps-dev): bump shell-quote from 1.8.3 to 1.8.4 in /datafusion/wasmtest/datafusion-wasm-app #22856 (dependabot[bot])
  • refactor(hash-aggr): Forward port the soft limit optimization to the new hash aggregation impl #22824 (2010YOUY01)
  • refactor(physical-plan): extract make_group_column factory + eager init at try_new + tighten Time variants #22751 (zhuqi-lucas)
  • minor: handle NULL array input in array_remove and array_replace #22790 (lyne7-sc)
  • Add sort tpch SQL benchmark #22814 (Omega359)
  • chore: Update to arrow/parquet 59.0.0 #22744 (alamb)
  • Upgrade minimal tokio-postgres version to address security advisory #22937 (AdamGS)
  • Clearly gate sliding SUM(DISTINCT) type support #22866 (kumarUjjawal)
  • refactor: introduce ProbeEnd state in NestedLoopJoinExec #22865 (nathanb9)
  • refactor: Simplify heap size estimation for types that own no heap allocations #22918 (mkleen)
  • refactor(hash-aggr): Migrate existing tests on GroupsHashAggregateStream #22953 (2010YOUY01)
  • Include null_aware status in the relevant Join node display implementations #22913 (AdamGS)
  • chore(deps): bump pyjwt from 2.12.0 to 2.13.0 #22966 (dependabot[bot])
  • ci: Setup valid Cargo.lock for depcheck to unblock CI #22933 (AdamGS)
  • chore(deps-dev): bump launch-editor from 2.10.0 to 2.14.1 in /datafusion/wasmtest/datafusion-wasm-app #22970 (dependabot[bot])
  • chore(deps): bump cryptography from 46.0.7 to 48.0.1 #22968 (dependabot[bot])
  • refactor: Simplify heap size estimation for arrays #22954 (mkleen)
  • Remove orphaned snowflake_flatten_validation.sql script #22938 (AdamGS)
  • chore(deps): bump insta-cmd from 0.6.0 to 0.7.0 #22976 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.81.8 to 2.81.11 #22973 (dependabot[bot])
  • chore(deps): bump prost-build from 0.14.3 to 0.14.4 #22843 (dependabot[bot])
  • Add .gitignore for proto-models #22977 (Jefffrey)
  • Fix leaf expression reconciliation #22971 (cetra3)
  • Make LogicalPlan::Unnest expression/rebuild contracts consistent #22783 (nathanb9)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 6 updates #22975 (dependabot[bot])
  • Refactor outer join null-rejection analysis to track join sides directly #22870 (kosiew)
  • chore: attach Diagnostic to unary operator type errors #21288 (hcrosse)
  • refactor: make scalar distance u64 and overflow aware #22892 (sweb)
  • bugfix: changed return type of spark's width_bucket to i64 #22811 (aguilaredu)
  • chore(deps-dev): bump webpack-dev-server from 5.2.4 to 5.2.5 in /datafusion/wasmtest/datafusion-wasm-app #23009 (dependabot[bot])
  • Add sorted TopK TPC-H benchmark target #23003 (geoffreyclaude)
  • test: correct feature gating of two datafusion-common tests #23044 (Phoenix500526)
  • test: gate hash-dependent approx_distinct tests behind not(force_hash_collisions) #23053 (Phoenix500526)
  • Skip loading Parquet page index when row-group statistics already prove it cannot prune #22857 (RatulDawar)
  • Return errors on string builder offset overflow in replace and initcap #22990 (kosiew)
  • minor: reuse ColumnarValue::into_array in map's expand_if_scalar and avoid uncessary clones #22984 (nathanb9)
  • refactor: add try_to_proto / try_from_proto to DynamicFilterPhysicalExpr #22452 (AnuragRaut08)
  • minor: Validate batch_size configuration when setting it #23054 (2010YOUY01)
  • Fix shared TopK early exit with shared prefix threshold #22991 (geoffreyclaude)
  • bench: add correlated-proxy case to the predicate_eval suite #22919 (adriangb)
  • refactor: name build-row and matchable-map presence checks in hash join #23024 (Phoenix500526)
  • IN LIST: clean up generic static filtering #21927 (geoffreyclaude)
  • test: drive stdin store reuse through get_or_create #23061 (huan233usc)
  • test: Move default cache tests to default cache file #23040 (mkleen)
  • Optimize Parquet row-filter struct schema pruning #22960 (shehab-ali)
  • Fix DuckDB unparse for optimized join projections #23002 (goutamadwant)
  • chore: gate internal_datafusion_err import behind the proto feature #23075 (Phoenix500526)
  • Perf: avoid redundant comparison in SortPreservingMerge round-robin tie-breaker; optimize inner loop #23107 (Dandandan)
  • Move Parquet input_file_name() tests to input_file_name.slt #23123 (AdamGS)
  • chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 #23115 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.81.11 to 2.82.2 #23114 (dependabot[bot])
  • [sql]: remove deprecated TableReference re-exports #23102 (mgkz0)
  • chore: cargo update -p quinn to resolve security audit issue #23122 (Jefffrey)
  • refactor(hash-aggr): Use EmitTo to output #23055 (2010YOUY01)
  • refactor: centralize TopK heap boundary handling #23091 (kumarUjjawal)
  • IN LIST: add UInt8 bitmap filter #23011 (geoffreyclaude)
  • chore(physical-plan): remove deprecated RowIndex struct (Closes #23080 - partial) #23143 (Dodothereal)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 5 updates #23118 (dependabot[bot])
  • Fix projection functional dependency remapping #23028 (hhhizzz)
  • Migrate case conversion and substr_index to fallible string view builder APIs #23074 (kosiew)
  • chore: use Vec instead of OffsetBuilder #23195 (comphead)
  • Fix final hash aggregate output regression by materializing once #23182 (hhhizzz)
  • Add regression coverage for quoted dotted column aliases #23155 (kosiew)
  • feat(functions-aggregate): support sum(interval) #23177 (SubhamSinghal)
  • chore(deps): bump itertools from 0.14.0 to 0.15.0 #23119 (dependabot[bot])
  • refactor: centralize join-input table-scan filter extraction before u… #23166 (Phoenix500526)
  • refactor: factor distinct-from unparsing into a shared helper #23163 (Phoenix500526)
  • IN LIST: unify bitmap filter implementations #23035 (geoffreyclaude)
  • Avoid repeated EmitTo::First in partial hash aggregate output #23250 (hhhizzz)
  • Fix metrics for repartition when preserve_order=true #20924 (xanderbailey)
  • chore(deps): bump taiki-e/install-action from 2.82.2 to 2.82.6 #23254 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group with 5 updates #23256 (dependabot[bot])
  • refactor: make_map_batch array handling #23228 (nathanb9)
  • bench(hj): Add missing Q16–Q23 to benchmarks #23257 (LiaCastaneda)
  • Restrict trigger push branch for GitHub Workflow #23278 (apupier)
  • Aggregations Support Partitioning::Range #23239 (gene-bordegaray)
  • refactor(hash-aggr): Migrate ordered partial/final aggregation #23181 (2010YOUY01)
  • Fix CI failure by Ignore quick-xml audit advisories #23298 (alamb)
  • refactor(hash-aggr): Migrate partial-reduce hash aggregation #23233 (2010YOUY01)
  • fix(EnsureRequirements): remap sort requirement through ProjectionExec on pushdown #23199 (Jeadie)
  • chore(deps): bump cmov from 0.5.3 to 0.5.4 #23300 (dependabot[bot])
  • Minor: Make BloomFilterStatistics and RowGroupAccessPlanFilter::prune_by_bloom_filters public #23302 (xudong963)
  • Add basic sql benchmark runner for running sql benchmarks #23052 (Omega359)
  • spark: support collect_list collect_set for windows execution #23281 (comphead)
  • chore: extend pre commit instructions for AI agents #23313 (comphead)
  • chore: add Cargo http options to handle download errors #23314 (comphead)
  • Fix inexact partitioned TopK sort pushdown #23301 (xudong963)
  • Add IN list sqllogictest test (and integer type coverage) #23305 (alamb)
  • bench: add array_has array-needle benchmarks #23335 (freakyzoidberg)
  • Add regression tests for hash-join dynamic filter expression policy #23319 (kosiew)
  • chore: update crossbeam-epoch to 0.9.20 #23358 (Phoenix500526)
  • chore(docs): resolve some docs typos #23347 (devanbenz)
  • IN LIST: add Float16 bitmap filter #23311 (geoffreyclaude)
  • chore(deps): bump astral-sh/setup-uv from 8.2.0 to 8.3.1 #23366 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.82.6 to 2.82.10 #23365 (dependabot[bot])
  • chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 #23367 (dependabot[bot])
  • minor: rename aggregate stream modules to match contents #23372 (alamb)
  • test: cover float IN list predicates #23373 (alamb)
  • test: Add coverage for NOT IN predicates #23378 (alamb)
  • chore(deps): bump runs-on/action from 2.1.2 to 2.2.0 #23363 (dependabot[bot])
  • chore: Update to arrow/parquet 59.1.0 #23312 (alamb)
  • Fix:22477 any all schema error #22915 (HairstonE)
  • Push sort requirements through simple projections #23288 (aectaan)
  • refactor(hash-aggr): Simplify aggregate hash table with tempated functions #23324 (2010YOUY01)
  • refactor: centralizing shared-allocation accounting for Arc DFHeapSize impls #23349 (saadtajwar)
  • Fix union equivalence schema rewrite with stale constants #23375 (xudong963)
  • Fix memory size accounting for grouped median and avg #23357 (lyne7-sc)
  • chore(spm): extract initialize all parititions helper #23419 (rluvaton)
  • refactor: extract parquet projection read plan into its own module #23396 (adriangb)
  • Perf: Add short circuit for primitive vectorized equal_to #23343 (Rich-T-kid)
  • refactor: centralize date_bin per-row mapping #23034 (kumarUjjawal)
  • chore: cleanup some TODO items in sqllogictests #23382 (Jefffrey)
  • bench: add date_part benchmark #23350 (theirix)
  • refactor: de-duplicate parquet read plan construction #23426 (adriangb)
  • chore(deps): bump soupsieve from 2.8.3 to 2.8.4 #23432 (dependabot[bot])
  • Use concat_elements_dyn from arrow-rs #23211 (pepijnve)
  • perf(physical-plan): fold PlanProperties fast-path into with_new_children_if_necessary (PR 1 of #22555) #23332 (zhuqi-lucas)
  • Minor: Fix docs for JoinSet #23448 (alamb)
  • test: add Poll::Pending spill stream coverage for async spill re-entry paths #23353 (pantShrey)
  • Test: add more aggregation focused dictionary sql logic test #23280 (Rich-T-kid)
  • minor: Remove .gitignore item for datafusion-examples #23409 (2010YOUY01)
  • minor: remove local file commited by mistake #23476 (2010YOUY01)
  • bench: add sort benchmarks for various data profile #23346 (rluvaton)
  • refactor(hash-aggr): Migrate single mode hash aggregation #23408 (2010YOUY01)
  • test: add sqllogictest coverage for DISTINCT / GROUP BY / aggregation on map columns #23406 (PG1204)
  • chore: use new OffsetBuffer::subtract helper #23424 (rluvaton)
  • Decode Hive partition values in listing tables #23226 (yinli-systems)
  • ci: Use install-action instead of cargo install to speed up CI #23477 (2010YOUY01)
  • Add minimal genarator-like stream implementation #23530 (pepijnve)
  • chore(deps): bump actions/stale from 10.3.0 to 10.4.0 #23557 (dependabot[bot])
  • chore(deps): bump actions/labeler from 6.1.0 to 6.2.0 #23556 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.82.10 to 2.83.2 #23555 (dependabot[bot])
  • chore(deps): bump astral-sh/setup-uv from 8.3.1 to 8.3.2 #23554 (dependabot[bot])
  • chore(deps): bump actions/setup-node from 6 to 7 #23550 (dependabot[bot])
  • perf(physical-expr): cache remapped expression in DynamicFilterPhysicalExpr::current() #23532 (zhuqi-lucas)
  • Preserve string slice function return types #23330 (xudong963)
  • Allow Range partitioned inputs to PartitionedTopK #23355 (stuhood)
  • chore: Simplifying SortPreservingMergeStream to use generators instead of state machine #23407 (rluvaton)
  • Enforce co-partitioning for sort merge and symmetric hash joins #23480 (gene-bordegaray)
  • Infer placeholder type from ANY/ALL subquery, unit tests #22545 (HairstonE)
  • Use octet_length for ClickBench Q27/Q28 byte-length semantics #23475 (kosiew)
  • chore: group codeql action dependabot updates #23561 (Jefffrey)
  • chore(deps): bump the codeql-actions group with 2 updates #23610 (dependabot[bot])
  • minor: validate config recursion_limit when setting it #23592 (2010YOUY01)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 10 updates #23613 (dependabot[bot])
  • Fix within group aggregates with unparser #22195 (cetra3)
  • bench(sort): fix sort axis benchmark run on single partition #23614 (rluvaton)
  • minor: validate config max_spill_file_size_bytes when setting it #23594 (2010YOUY01)
  • minor: validate config soft_max_rows_per_output_file when setting it #23597 (2010YOUY01)
  • chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /datafusion/wasmtest/datafusion-wasm-app #23625 (dependabot[bot])
  • chore(deps): bump serde_with from 3.18.0 to 3.21.0 #23624 (dependabot[bot])
  • minor: validate config minimum_parallel_output_files when setting it #23596 (2010YOUY01)
  • minor: validate config meta_fetch_concurrency when setting it #23595 (2010YOUY01)
  • try parallel ci #23618 (blaginin)
  • allow interleaveExec to support Range partioning #23623 (Rich-T-kid)
  • Support UDTFs in information_schema.routines / SHOW FUNCTIONS #23438 (zhuqi-lucas)
  • Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat #23071 (gstvg)
  • minor(CI): Use install-action to speed up ci #23661 (2010YOUY01)
  • bench: add FixedSizeBinary coverage to multi_group_by benchmark #23650 (alamb)
  • test: Fix malformed regexp_instr error tests and add slt coverage #23620 (alamb)
  • Mark null-propagating math functions as strict #23527 (lyne7-sc)
  • Fix ordering for UNION ALL over heterogeneous constants #23528 (vadimpiven)
  • chore: downsize sql_planner_extended logical_plan_optimize sample size to 5 #23659 (Jefffrey)
  • test: add advanced dictionary test #23483 (Rich-T-kid)
  • bench: parquet scan with a table schema narrower than a nested column #23397 (adriangb)
  • Cap SortPreservingMerge statistics by fetch #23359 (discord9)
  • array_agg() add tests and benchmarks #23740 (fred1268)
  • test: More slt tests for iszero function #23713 (2010YOUY01)
  • feat(physical-plan): Allow co-partitioned Partitioning::Range inputs for left-side hash joins #23487 (JSOD11)
  • chore(deps-dev): bump webpack-dev-server from 5.2.5 to 5.2.6 in /datafusion/wasmtest/datafusion-wasm-app #23768 (dependabot[bot])
  • chore(deps): bump actions/checkout from 7.0.0 to 7.0.1 #23746 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.82.6 to 2.84.0 #23748 (dependabot[bot])
  • chore(deps): bump actions/labeler from 6.2.0 to 7.0.0 #23749 (dependabot[bot])
  • chore(deps): bump codecov/codecov-action from 5.5.5 to 7.0.0 #23750 (dependabot[bot])
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 15 updates #23771 (dependabot[bot])
  • chore: fix SlidingDistinctCountAccumulator::size() to include budget for distinct values #23399 (comphead)
  • test: improve md5 function SQL test coverage #23757 (2010YOUY01)
  • chore(deps-dev): bump fast-uri from 3.1.2 to 3.1.4 in /datafusion/wasmtest/datafusion-wasm-app #23778 (dependabot[bot])
  • chore(deps-dev): bump shell-quote from 1.8.4 to 1.10.0 in /datafusion/wasmtest/datafusion-wasm-app #23769 (dependabot[bot])
  • test: improve isnan function SQL test coverage #23754 (2010YOUY01)
  • chore(deps): bump the codeql-actions group with 2 updates #23745 (dependabot[bot])
  • test: improve digest function SQL test coverage #23756 (2010YOUY01)
  • test: improve sha function SQL test coverage #23758 (2010YOUY01)
  • test: improve lcm function SQL test coverage #23755 (2010YOUY01)
  • allow range to satisfy key distribution generally #23680 (gene-bordegaray)
  • fix: do not treat concat as preserving lexicographical ordering #23804 (buraksenn)
  • refactor(proto): delegate deprecated ProjectionExec serde shims to new hooks #23731 (adriangb)
  • Add setter for TaskContext::task_id #23837 (pepijnve)
  • refactor(hash-aggr): Support spilling for ordered aggregation #23657 (2010YOUY01)
  • Unwrap widening Date32 -> Date64 casts in comparison predicates #23729 (adriangb)
  • test (slt): add memory-limited aggregation sqllogictests #23838 (naman-modi)
  • chore(deps-dev): bump ws from 8.18.2 to 8.21.1 in /datafusion/wasmtest/datafusion-wasm-app #23866 (dependabot[bot])
  • chore(deps-dev): bump http-proxy-middleware from 2.0.9 to 2.0.10 in /datafusion/wasmtest/datafusion-wasm-app #23865 (dependabot[bot])
  • test: add functional_dependencies.slt covering functional dependency driven optimizations #23821 (alamb)
  • chore(deps-dev): bump webpack-dev-server from 5.2.6 to 6.0.0 in /datafusion/wasmtest/datafusion-wasm-app #23868 (dependabot[bot])
  • refactor(unparser): centralize aggregate-scope rendering in the SQL unparser #23789 (naman-modi)
  • Add FixedSizeList support for recursive struct schema adaptation #22980 (kosiew)
  • test: cover array_agg(DISTINCT) on dictionaries and bounded retract_batch memory #23873 (alamb)
  • chore: adjust size accounting for min_max #23899 (comphead)
  • Add ObjectStore-backed TempFileFactor / spill example #23170 (alamb)
  • Various ScalarValue numeric method fixes & refactors (especially decimal) #23631 (Jefffrey)
  • chore: simplify SortPreservingMergeStream to be as textbook-like as possible #23702 (rluvaton)
  • chore: Squelch "unused code" warning #23924 (neilconway)
  • Add name filter to metrics #23719 (gabotechs)
  • chore(deps): bump the codeql-actions group with 2 updates #23938 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.84.0 to 2.85.2 #23941 (dependabot[bot])
  • chore(deps): bump actions/stale from 10.4.0 to 11.0.0 #23942 (dependabot[bot])
  • chore(deps): bump base64 from 0.22.1 to 0.23.0 #23944 (dependabot[bot])
  • chore(deps): bump astral-sh/setup-uv from 8.3.2 to 9.0.0 #23939 (dependabot[bot])
  • chore: refactor SortMergeJoin bitwise stream to generators and simplify to be textbook like as possible #23761 (rluvaton)
  • refactor: address review feedback on percentile_cont(DISTINCT) accumulator #23946 (viirya)
  • chore: remove unused header file #23958 (Jefffrey)
  • Fill in missing utf8view support in function type coercion #23916 (Jefffrey)
  • chore: refactor VarianceAccumulator, add tests and benchmark #23977 (neilconway)
  • minor(test): cover partially ordered aggregate spilling #23947 (buraksenn)
  • chore(ordered-partial-aggregate): move OrderedPartialAggregateStream to generators for readability #23951 (rluvaton)
  • test: improve round sqllogictest coverage #23973 (2010YOUY01)
  • test: improve gcd sqllogictest coverage #23972 (2010YOUY01)
  • test: improve rpad sqllogictest coverage #23968 (2010YOUY01)
  • test: improve lpad sqllogictest coverage #23969 (2010YOUY01)
  • Add benchmarks for hashjoin candidate equality filtering #23980 (shehab-ali)
  • Optimize Spark hex null handling #23688 (floze-the-genius)
  • fix(proto): prevent duplicate partition statistics on roundtrip #23999 (buraksenn)
  • Report peak MemoryPool reservation per query in benchmarks #23985 (adriangb)
  • refactor(proto): move PartitionedFile / FileGroup serde into datafusion-datasource #24006 (adriangb)
  • chore: refactor MaterializingSortMergeJoinStream into generators and simplify code to be textbook like as possible #23976 (rluvaton)
  • refactor(proto): put Partitioning / sort-expression serde on the types #24003 (adriangb)
  • bench: use seedable rng for reproducibility #23653 (theirix)
  • test: Fix data_pagesize_limit extraction in parquet writer props roundtrip test #23664 (jackylee-ch)
  • bench: extend BoundedWindowAggExec many-partitions benchmark #24032 (neilconway)
  • refactor: move arrow integer hex dispatch to datafusion-common #23917 (buraksenn)
  • test: add IN list slt coverage for temporal, Decimal128 and Interval types #23875 (alamb)
  • chore: rows_to_array cleanup for expecting single field #24040 (saadtajwar)
  • minor: Add slt test for nullable window retract #24025 (2010YOUY01)
  • WindowTopN dense_rank benchmark #24050 (SubhamSinghal)
  • minor(fix): correct to_date results for formatted pre-epoch datetimes #24049 (buraksenn)
  • minor(test): strengthen sort-merge join spilling coverage #23988 (buraksenn)
  • refactor(hash-aggr): Support spilling for single mode aggregation #23965 (2010YOUY01)
  • test: improve find_in_set sqllogictest coverage #23970 (2010YOUY01)
  • IN LIST: isolate branchless filter implementation #23907 (geoffreyclaude)
  • bench: add nested-type (List/Struct/Map) cases to first_value/last_value benchmark #24075 (zhuqi-lucas)
  • chore(deps): bump taiki-e/install-action from 2.85.2 to 2.85.6 #24081 (dependabot[bot])
  • chore(deps): bump the codeql-actions group with 2 updates #24080 (dependabot[bot])
  • bench: add ArrowBytesMap benchmarks #24078 (Punisheroot)
  • chore(deps): bump cryptography from 48.0.1 to 50.0.0 #24091 (dependabot[bot])
  • fix(physical-plan): preserve Exact(0) in FilterExec for null_count, distinct_count and total_byte_size upon empty input #24000 (asolimando)
  • chore: cleanup OrderedPartialAggregateStream more #24012 (rluvaton)
  • chore: apply workspace lints to all crates #24076 (emilk)
  • feat(functions-aggregate): support nested types (List, Struct, Map) in first_value / last_value GroupsAccumulator #23628 (zhuqi-lucas)
  • Add config-matrix tests in enforce_distribution.rs for range-satisfaction settings #23627 (blinding-pixels)
  • chore(deps-dev): bump fast-uri from 3.1.4 to 3.1.5 in /datafusion/wasmtest/datafusion-wasm-app #24092 (dependabot[bot])
  • Add support for running sql benchmarks with command line arguments #23772 (Omega359)
  • refactor(hash-aggr): Support spilling for partial and final mode aggregation #24061 (2010YOUY01)
  • Proto: add DataSink serialization hook #23752 (Phoenix500526)
  • bench: multi-conjunct shared-prefix struct row-filter pushdown #23524 (SubhamSinghal)
  • Preserve grouping ID during aggregate CSE #24144 (notfilippo)
  • Add DataSource/FileSource proto hooks and FileScanConfig serde #23683 (kumarUjjawal)
  • tests: add SLT test coverage for MERGE INTO #24174 (alamb)
  • chore: add runendencoded & listview types to dfschema equality methods #24138 (Jefffrey)
  • refactor(proto): destructure plan and proto structs in aggregate and window serde hooks #24166 (adriangb)
  • chore(deps): bump the all-other-cargo-deps group across 1 directory with 10 updates #24163 (dependabot[bot])
  • test: add UnionArray hashing SQL coverage. #24199 (VaibhaveS)
  • fix(physical-plan): count empty grouping sets in the aggregate row estimate for an empty input #24039 (asolimando)
  • test(proto): add missing physical plan round-trip coverage #24172 (adriangb)
  • test(proto): split roundtrip_physical_plan.rs by plan category #24223 (adriangb)
  • physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time #24094 (dariocurr)
  • Parquet row filter struct access tree #23217 (SubhamSinghal)
  • Fix aggregate accumulator capacity accounting #24099 (kosiew)
  • refactor: Refactor numeric sign and padding in Spark format_string #24115 (JSOD11)
  • chore(deps): bump the all-other-cargo-deps group with 4 updates #24254 (dependabot[bot])
  • chore(deps): bump taiki-e/install-action from 2.85.6 to 2.85.10 #24253 (dependabot[bot])
  • chore(deps): bump runs-on/action from 2.2.0 to 2.3.0 #24252 (dependabot[bot])
  • chore(deps): bump Swatinem/rust-cache from 2.9.1 to 2.9.2 #24251 (dependabot[bot])
  • Add FixedSizeBinary support for MultiGroupBy #23646 (maxburke)
  • refactor: make apply_expression_roots more ergonomic #24226 (jayshrivastava)
  • chore(deps): bump toml from 0.9.12+spec-1.1.0 to 1.1.3+spec-1.1.0 #24256 (dependabot[bot])
  • refactor: moving WindowTopN before EnsureRequirements #24191 (saadtajwar)
  • chore(deps): bump the codeql-actions group with 2 updates #24250 (dependabot[bot])
  • [branch-55] Prepare for 55 release - version number, changelog #24292 (timsaucer)
  • [branch-55] Update changelog #24314 (timsaucer)
  • [branch-55] fix: correct list field inner type in array functions (#24345) #24367 (timsaucer)
  • [branch-55] fix wrong TopK results from re-reading already-delivered row groups (#24352) #24368 (zhuqi-lucas)
  • [branch-55]: don't runtime-prune row groups while a page-index RowSelection is live (#24355) #24374 (zhuqi-lucas)
  • [branch-55] fix: preserve the input list's inner field in array_append/prepend/replace - #24365 #24377 (timsaucer)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

    80	dependabot[bot]
    46	Neil Conway
    42	Yongting You
    41	Adrian Garcia Badaracco
    37	Andrew Lamb
    27	Burak Şen
    24	Phoenix
    24	linfeng
    22	Andy Grove
    19	Kumar Ujjawal
    19	Michael Kleen
    18	Raz Luvaton
    17	Qi Zhu
    16	Adam Gutglick
    14	Bruce Ritchie
    14	Giorgio Maria Federico Birnthaler
    13	Jeffrey Vo
    12	Geoffrey Claude
    12	Oleks V
    12	xudong.w
    10	Gene Bordegaray
    10	Saad Tajwar
    10	Subham Singhal
    10	Tim Saucer
    10	kosiew
     9	Nathan
     9	theirix
     8	Huaijin
     6	Daniël Heres
     6	Zhen Chen
     5	Alessandro Solimando
     5	Amogh Ramesh
     5	Ariel Miculas-Trif
     5	Jayant Shrivastava
     5	Liang-Chi Hsieh
     5	Lía Adriana
     5	Matt Butrovich
     5	RIchard Baah
     5	kid
     4	Brent Gardner
     4	Goutam Adwant
     4	H
     4	Megakaizo
     4	Nuno Faria
     4	Pepijn Van Eeckhoudt
     4	Sean Kenneth Doherty
     4	Varun
     4	Xuanyi Li
     4	chakkk309
     4	discord9
     3	Alex Metelli
     3	ByteBaker
     3	Huang Qiwei
     3	Justin O'Dwyer
     3	Matthew Patton
     3	Mithun Chicklore Yogendra
     3	Moe
     3	Shehab Ali
     3	Simon Vandel Sillesen
     3	Xin Huang
     3	Yin Li
     3	crm26
     3	gstvg
     3	pantShrey
     3	pchintar
     2	Anurag Tryambak Raut
     2	Bert Vermeiren
     2	Bhargava Vadlamani
     2	David López
     2	Diego Perez Giordán
     2	Edson Petry
     2	EeshanBembi
     2	Emily Matheys
     2	Filip Petkovski
     2	Florian Müller
     2	Ford
     2	Fred Thomas
     2	Gabriel
     2	Guocheng(Eric) Song
     2	JS
     2	Kanishk Sachan
     2	Karpagam Balasubramaniam
     2	Krishna Sudarshan J
     2	Louis Vialar
     2	Matthew Kim
     2	Nagato Yuzuru
     2	Naman Modi
     2	Peter L
     2	Peter Lee
     2	Pierre Lacave
     2	Prateek Ganigi
     2	Puneet Dixit
     2	Tobias Schwarzinger
     2	WeblWabl
     2	Zac Farrell
     2	Zeel Rajodiya
     2	dario curreri
     2	fys
     2	jackylee
     2	jj.lee
     2	nanookclaw
     1	7. Sun
     1	Ahmed EL.
     1	Asish Kumar
     1	Aurélien Pupier
     1	Ben Chambers
     1	Braedon Wooding
     1	Brijesh Thakkar
     1	Bruno Volpato
     1	Bukhtawar Khan
     1	Daipayan Mukherjee
     1	DevShiba
     1	Dmitrii Blaginin
     1	Eduardo Aguilar
     1	Egor Markov
     1	Emil Ernerfeldt
     1	Evgeniy Mineev
     1	Filippo
     1	Floze
     1	Georgi Krastev
     1	Gunther Xing
     1	Gustavo Schneiter
     1	Harrison Crosse
     1	Haseeb Nazir
     1	Jack Eadie
     1	Jason Wong
     1	Jordan Epstein
     1	Joseph Lenton
     1	Kazantsev Maksim
     1	Kent Wu
     1	Kristin Cowalcijk
     1	Krisztián Szűcs
     1	Lavkesh Lahngir
     1	Lining Pan
     1	Ma Zhengxuan
     1	Marc Brinkmann
     1	Marko Milenković
     1	Matt Van Horn
     1	Max Burke
     1	Minh Vu
     1	Nam2ee
     1	Namgung Chan
     1	Nathan Bezualem
     1	Pablo Abad Rubio
     1	Pavan51
     1	Ratul Dawar
     1	Recoordinate
     1	Ruchir Tripathi
     1	RyanStewart
     1	Sai Asish Y
     1	Savan Nahar
     1	Sergei Grebnov
     1	Sergey Zhukov
     1	Stu Hood
     1	Thomas Santerre
     1	Tian Teng
     1	Vadim Piven
     1	VaibhaveS
     1	Victorien
     1	Vikrant Mehta
     1	Vismay
     1	Wenqi Mou
     1	Xander
     1	Xuanwo
     1	Yonatan Striem Amit
     1	Zhen-Lun (Kevin) Hong
     1	ajegou
     1	blinding-pixels
     1	eliot1480
     1	jeroenflvr
     1	kkrainov
     1	subotac
     1	yoongbok lee
     1	zhengpeng
     1	zhigang

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.