dev/changelog/48.0.0.md
This release consists of 267 commits from 89 contributors. See credits at the end of this changelog for more information.
Breaking changes:
flatten so it does only a level, not recursively #15160 (delamarch3)simplify_expressions rule #15735 (xudong963)ExecutionPlan::partition_statistics API #15852 (xudong963)ParquetExec #15973 (miroim)ArrowExec #16006 (miroim)MemoryExec #16007 (miroim)JsonExec #16005 (miroim)Filter::having field #16154 (findepi)VARCAHR from Utf8 to Utf8View #16142 (zhuqi-lucas)Expr struct #16207 (hendrikmakait)Performance related:
optimize_projection #15746 (xudong963)optimize_projection by improving is_projection_unnecessary #15761 (xudong963)optimize_projection #15787 (xudong963)GroupsAccumulator for Avg duration #15748 (shruti2522)string::ascii function #16087 (tlm365)Implemented enhancements:
datafusion-spark crate #15168 (shehabgamin)BaselineMetrics to generate_series() table function #16255 (2010YOUY01)Fixed bugs:
replace_order_preserving_variants method during EnforceDistribution optimizer #15808 (xudong963)EnforceSorting optimizer (two places) #15822 (xudong963)map_values return type #15901 (rluvaton)build_predicate_expression method doesn't process false expr correctly #15995 (xudong963)Display for LogicalPlan::Projection #16164 (atahanyorganci)Documentation updates:
OPTIONS clause #15708 (marvelshan)from_unixtime function documentation #15844 (Viicos)union_tag scalar function #14687 (gstvg)FormatOptions to Config #15793 (blaginin)bloom_filter_on_read as a reading config #15933 (nuno-faria)ParquetExec, AvroExec, CsvExec, JsonExec #16034 (alamb)datafusion.execution.collect_statistics #16100 (alamb)SessionContext::register_parquet obey collect_statistics config #16080 (adriangb)TrackConsumersPool as default in datafusion-cli #16081 (ding-young)dataframe! example into dedicated example #16197 (comphead)Other:
Accumulator for avg duration #15468 (shruti2522)aggregate.slt #15786 (xudong963)or_fun_call and unnecessary_lazy_evaluations lints on core #15807 (Rachelint)UNION for distinct results #15814 (phillipleblanc)MemoryPool::memory_limit to expose setting memory usage limit #15828 (Rachelint)ILIKE expression support in SQL unparser #15820 (ewgenius)Diagnostic easy/convinient to attach by using macro and avoiding map_err #15796 (logan-keede)plan_with_order_breaking_variants method #15842 (xudong963)CoalescePartitionsExec proto serialization #15824 (lewiszlw)CoalescePartitionsExec::with_fetch method #15849 (lewiszlw)No UDF registered error #15843 (comphead)dbg! #15858 (phillipleblanc)datafusion.execution.parquet.coerce_int96 setting #15723 (alamb)ListingTable / ListingTableOptions docs #15767 (alamb)FileSource, DataSource and DataSourceExec #15766 (alamb)flatten scalar function when inner list is FixedSizeList #15898 (gstvg)evaluate_bounds #15716 (davidhewitt)UNNEST projection with the table column alias #15879 (goldmedal)character_length #15931 (Dandandan)rowsort to slt test #15942 (xudong963)unwrap_or with unwrap_or_else for improved lazy… #15841 (NevroHelios)Reuse rows in row cursor stream #15913 (acking-you)INFORMATION_SCHEMA #15965 (goldmedal)AvroExec #15987 (miroim)CsvExec #15991 (miroim)should_enable_page_index from mod.rs to opener.rs #16026 (miroim)EXPLAIN #16043 (tlm365)return_field_from_args #16039 (alamb)MIN and MAX for DataType::List #16025 (gabotechs)char expression #15994 (andygrove)collect_statistics #16098 (alamb)fuzz tests #16111 (alamb)ScalarFunctionArgs::return_type method #16113 (alamb)contains function expression #16046 (liamzwbao)params.rs #16141 (liamzwbao)Accumulator::return_type and StateFieldsArgs::return_type to help with upgrade to 48 #16112 (alamb)pool.rs to use insta #16145 (lifan-ake)unproject_sort_expr to handle arbitrary expressions #16127 (phillipleblanc)logical_plan tests to insta #16184 (lifan-ake)AggregateExec.partition_statistics #15954 (UBarney)RepartitionExec #16093 (gabotechs)do_append_val_inner #16201 (liamzwbao)--substrait-round-trip option in sqllogictests #16183 (gabotechs)pub use PruningStatistics #16264 (alamb)Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.
30 dependabot[bot]
29 Andrew Lamb
16 xudong.w
14 Adrian Garcia Badaracco
10 Chen Chongchen
8 Gabriel
8 Oleks V
7 miro
6 Tommy shu
6 kamille
5 Lokesh
5 Tim Saucer
4 Dmitrii Blaginin
4 Jay Zhan
4 Nuno Faria
4 Yongting You
4 logan-keede
3 Christian
3 Daniël Heres
3 Liam Bao
3 Phillip LeBlanc
3 Piotr Findeisen
3 ding-young
2 Andy Grove
2 Atahan Yorgancı
2 Brayan Jules
2 Georgi Krastev
2 Jax Liu
2 Jérémie Drouet
2 LB7666
2 Leonardo Yvens
2 Qi Zhu
2 Sergey Zhukov
2 Shruti Sharma
2 Tai Le Manh
2 aditya singh rathore
2 ake
2 cht42
2 gstvg
2 kosiew
2 niebayes
2 张林伟
1 Ahmed Mezghani
1 Alexander Droste
1 Andy Yen
1 Arka Dash
1 Arttu
1 Dan Harris
1 David Hewitt
1 Davy
1 Ed Seidl
1 Eshed Schacham
1 Evgenii Khramkov
1 Florent Monjalet
1 Galim Bikbov
1 Garam Choi
1 Hamir Mahal
1 Hendrik Makait
1 Jonathan Chen
1 Joseph Fahnestock
1 Kevin Zimmerman
1 Lordworms
1 Lía Adriana
1 Matt Butrovich
1 Namgung Chan
1 Nelson Antunes
1 Patrick Sullivan
1 Raz Luvaton
1 Ruihang Xia
1 Ryan Roelke
1 Sam Hughes
1 Shehab Amin
1 Sile Zhou
1 Simon Vandel Sillesen
1 Tom Montgomery
1 UBarney
1 Victorien
1 Xiangpeng Hao
1 Zaki
1 chen quan
1 delamarch3
1 discord9
1 hozan23
1 irenjj
1 jsai28
1 m09526
1 suibianwanwan
1 the0ninjas
1 wiedld
Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.