dev/changelog/42.0.0.md
This release consists of 296 commits from 73 contributors. See credits at the end of this changelog for more information.
Breaking changes:
datafusion-expr and remove logical expressions requirement for creating physical aggregate expression #11845 (jayzhan211)datafusion.catalog.has_headers default value set to true #11919 (korowa)schema_name to create the physical_name #11977 (joroKr21)LogicalPlan by value from TableProvider::get_logical_plan #12113 (askalt)Expr::Sort) #12177 (findepi)AggregateOptions struct and scalar_update_factor config setting #12241 (jc4x4)nth_value UDAF expr function #12279 (jcsherin)Result from return type in NamePreserver #12358 (jonahgao)Performance related:
Implemented enhancements:
FixedSizedList in array_distance function #12381 (Weijun-H)Fixed bugs:
TypeCoercion #11961 (jonahgao)bool_and #12017 (jonahgao)nth_value function #12076 (Weijun-H)supports_retract_batch to false for ApproxPercentileAccumulator #12132 (jonahgao)Documentation updates:
Accumulator::state docs #12053 (lewiszlw)datafusion-docs module compilation #12081 (alamb)EXPLAIN and EXPLAIN ANALYZE #12122 (devanbenz)array_distance function #12211 (austin362667)list_distinct alias link title #12246 (austin362667)map_keys & map_values for MAP type #12194 (dharanad)kurtosis_pop UDAF #12273 (goldmedal)CONTAINS function #12168 (tlm365)schema_force_string_view, rename config option to schema_force_view_types #12232 (wiedld)version() UDF #12429 (samuelcolvin)Other:
Duration vs Interval comparisons and Interval as LHS #11876 (samuelcolvin)regexp_like and regexp_match to work with StringView #11753 (alamb)crate:: imports in listing table provider module #11887 (findepi)Expr::display_name to Expr::schema_name, make UNNEST naming conform to convention #11797 (jayzhan211)CommonSubexprEliminate top-down like #11683 (peter-toth)generate_series tests for arrays #11921 (alamb)lit(true) and lit(false) more #11904 (alamb)approx_percentile_cont() aggregate function #11934 (2010YOUY01)datafusion-cli #11927 (PsiACE)LimitPushdown to physical-optimizer crate #11945 (lewiszlw)convert_to_state for AVG accumulator #11734 (alamb)Precision<usize> copy to make it clear clones are not expensive #11828 (alamb)CoalesceBatches to use an explicit state machine #11966 (berkaysynnada)REPEAT #11962 (tlm365)SessionStateBuilder::new_from_existing #11991 (goldmedal)min/max function to avoid crashes #12016 (tlm365)Accumulator::evaluate and Accumulator::state #12001 (lewiszlw).gitignore #12035 (alamb)ParquetExec / Parquet predicate pushdown #11994 (alamb)LimitedDistinctAggregation to physical-optimizer crate #12036 (Weijun-H)row_number to user-defined window function #12030 (jcsherin)sqlparser-rs v0.50.0 #12014 (samuelcolvin)datafusion-cli from Dockerfile #12018 (tlm365)coerce_plan_expr_for_schema #12051 (jonahgao)LATERAL subqueries #11456 (aalexandrov)clap 4.5.16 #12064 (findepi)AggregateUDFImpl::is_null #11989 (jayzhan211)BatchCoalescer to its own module #12047 (alamb)HAVING clause #12046 (goldmedal)dictionary_coercion to dictionary_comparison_coercion, add comments #12102 (alamb)StringArrayType trait #12027 (alamb)|| for StringViewArray #12063 (dharanad)CombinePartialFinalAggregate code readability #12128 (lewiszlw)LexRequirement alias as much as possible #12130 (lewiszlw)array_has avoid row converter for string type #12097 (jayzhan211)not_impl_error for approx_percentile_cont parameters validation #12133 (goldmedal)GroupedHashAggregateStream struct #12127 (2010YOUY01)array_has with eq kernel #12125 (jayzhan211)pr_comment.yml #12182 (alamb)AggregateExpr trait #12096 (lewiszlw)parse_vec_expr helper #12178 (findepi).../logical_plan/builder.rs #12196 (findepi)NamePreserver in SimplifyExpressions #12238 (jonahgao)AggregateFunctionExpr::field #12258 (lewiszlw)CombinePartialFinalAggregate rule into physical-optimizer crate #12167 (lewiszlw)Signature::Coercible #12275 (jayzhan211)struct and named_struct functions #11688 (Rafferty97)arrow/parquet 53.0.0, tonic, prost, object_store, pyo3 #12032 (alamb)TopKAggregation rule into physical-optimizer crate #12334 (lewiszlw)SortPreservingMergeExec #12302 (berkaysynnada)UnnestExec #12344 (joroKr21)ScalarValue::Binary* debug #12323 (alamb)array_has and inlist #12164 (jayzhan211)character_length() string function for ASCII-only case #12356 (2010YOUY01)DynamicFileCatalog in datafusion-catalog #11035 (goldmedal)SqlToRel::sql_expr_to_logical_expr_internal to reduce stack size #12384 (Jefffrey)spill_record_batch_by_size function #12389 (lewiszlw)null_equals_null case #12404 (Dandandan)substr() on StringView column's behavior is inconsistent with the old version #12383 (2010YOUY01)substr) #12433 (2010YOUY01)create temporary table #12439 (hailelagi)ListingOptions::with_file_extension_opt #12461 (alamb)array_has #12459 (samuelcolvin)Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.
39 Andrew Lamb
35 Piotr Findeisen
15 张林伟
10 Jax Liu
9 Jay Zhan
9 Jonah Gao
9 dependabot[bot]
8 wiedld
7 Chojan Shang
7 WeblWabl
7 Yongting You
6 Berkay Şahin
6 Eduard Karacharov
6 Tai Le Manh
6 kf zheng
5 Alex Huang
5 Bruce Ritchie
5 Lordworms
5 Samuel Colvin
4 Andy Grove
4 Dharan Aditya
4 HuSen
4 Huaijin
3 Arttu
3 Austin Liu
3 Daniël Heres
3 Dmitry Bugakov
3 Emil Ejbyfeldt
3 Georgi Krastev
3 JC
3 Oleks V
3 Trent Hauck
3 iamthinh
2 Athul T R
2 June
2 Liang-Chi Hsieh
2 Martin Hilton
2 Matt Green
2 Sergei Grebnov
2 Tim Saucer
2 Xiangpeng Hao
2 Xin Li
2 jcsherin
2 theirix
1 Adrian Garcia Badaracco
1 Albert Skalt
1 Alexander Alexandrov
1 Alexander Rafferty
1 Amey Chaugule
1 Cancai Cai
1 Dao Thanh Tung
1 Edmondo Porcu
1 FANNG
1 Haile
1 JasonLi
1 Jeffrey Vo
1 Leonardo Yvens
1 Maron Montano
1 Martin Kolb
1 Matthijs Brobbel
1 Michael J Ward
1 Mustafa Akur
1 Namgung Chan
1 Nick Cameron
1 Peter Toth
1 Phillip LeBlanc
1 Victor Barua
1 YjyJeff
1 mertak-synnada
1 ngli-me
1 peasee
1 waruto
1 yfu
Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.