dev/changelog/38.0.0.md
Breaking changes:
AggregateUDFImpl::accumulator signature, support ORDER BY for UDAFs #9874 (jayzhan211)OwnedTableReference and OwnedSchemaReference #9933 (comphead)Optimizer to use owned plans and TreeNode API (10% faster planning) #9948 (alamb)LogicalPlan::with_param_values #10016 (alamb)supports_zero_argument and avoid creating null array for empty args #10193 (jayzhan211)ScalarFunctionDefinition::Name #10277 (lewiszlw)Covariance (Sample) covar / covar_samp to be a User Defined Aggregate Function #10372 (jayzhan211)Performance related:
Arc<str> instead of Cow<&'a> in the analyzer #9824 (comphead)Implemented enhancements:
lower and upper functions #9971 (JasonLi-cn)unnest multiple arrays #10044 (jonahgao)DataFrame supports unnesting multiple columns #10118 (jonahgao)NestedLoopJoinExec #9676 (korowa)prefer_existing_union #10259 (NGA-TRAN)Fixed bugs:
WITH clause #9836 (jonahgao)unnest_generic_list handling of null list #9975 (jonahgao)RepartitionExec::execute #10009 (crepererum)RepartitionExec metrics #10025 (crepererum)in_list physical plans #10031 (advancedxy)joins.slt #10122 (korowa)substring function #10242 (jonahgao)Documentation updates:
COPY documentation to reflect changes #9754 (alamb)datafusion-federation to Integrations #9853 (phillipleblanc)AggregateUDFImpl::state_fields documentation #9919 (alamb)TreeNode #10035 (alamb)SessionState and SessionContext #10350 (alamb)Merged pull requests:
COPY documentation to reflect changes #9754 (alamb)Spice.ai to Known Users #9852 (phillipleblanc)datafusion-federation to Integrations #9853 (phillipleblanc)PARTITION BY clause, backwards compatible #9599 (MohamedAbdeen21)Atan2, Atan, Acosh, Asinh, Atanh to datafusion-function #9872 (Weijun-H)WITH clause #9836 (jonahgao)datafusion-function #9889 (Weijun-H)COPY ... OPTIONS (FORMAT..) options #9753 (alamb)spilled_rows metric to ExternalSorter by IPCWriter #9885 (erenavsarogullari)radians, signum, sin, sinh and sqrt functions to datafusion-functions crate #9882 (erenavsarogullari)Arc<str> instead of Cow<&'a> in the analyzer #9824 (comphead)struct instead of named_struct when there are no aliases #9897 (alamb)impl Into<Arc<str>> to create Arc<str> rather than &str #9916 (alamb)AggregateUDFImpl::accumulator signature, support ORDER BY for UDAFs #9874 (jayzhan211)AggregateUDFImpl::state_fields documentation #9919 (alamb)cbrt, cos, cosh, degrees to datafusion-functions #9938 (erenavsarogullari)OwnedTableReference and OwnedSchemaReference #9933 (comphead)AggregateExpr, PhysicalExpr and PhysicalSortExpr to physical-expr-core #9926 (jayzhan211)COUNT(1): Change the sentinel value's type for COUNT(*) to Int64 #9944 (gruuya)TableProvider::supports_filters_pushdown and remove deprecated function #9923 (alamb)AccumulatorArgs #9920 (alamb)CREATE EXTERNAL TABLE if table already exists. #9912 (MohamedAbdeen21)CommonSubexprEliminate #9959 (alamb)NamedStructField should be rewritten in OperatorToFunction #9968 (alamb)aggregate-functions #9960 (jayzhan211)Analzyer / check_plan #9974 (alamb)Expr::apply() instead of inspect_expr_pre() #9984 (peter-toth)unnest_generic_list handling of null list #9975 (jonahgao)DFSchema::index_of_column_by_name #9990 (lewiszlw)tree_node module #9995 (alamb)LogicalPlan::apply* and LogicalPlan::map* #9996 (alamb)PruningStatistics::row_counts and PruningStatistics::null_counts and make test match #10004 (alamb)LogicalPlan::clone() in LogicalPlan::map_children when possible #9999 (alamb)TreeNode::exists() API, avoid copying expressions #10008 (peter-toth)LogicalPlan::apply_subqueries and LogicalPlan::map_subqueries pub #9998 (alamb)RepartitionExec::execute #10009 (crepererum)Optimizer to use owned plans and TreeNode API (10% faster planning) #9948 (alamb)RepartitionExec metrics #10025 (crepererum)batch_size rather than batch_size-1 #10030 (JasonLi-cn)OptimizerRule::rewrite to rewrite in place, rewrite ExprSimplifier (20% faster planning) #9954 (alamb)LogicalPlan TreeNode methods #10037 (alamb)LogicalPlan::with_param_values #10016 (alamb)NamedStructField should be rewritten in OperatorToFunction in subquery regression (change ApplyFunctionRewrites to use TreeNode API #10032 (alamb)InlineTableScan via TreeNode API #10038 (alamb)crypto_expressions feature properly for --no-default-features #10059 (phillipleblanc)FileSinkExec to DataSinkExec #10065 (phillipleblanc)in_list physical plans #10031 (advancedxy)EliminateOuterJoin to implement OptimizerRule::rewrite() #10081 (peter-toth)lower and upper functions #9971 (JasonLi-cn)SHOW ALL in datafusion-cli #10079 (alamb)SessionConfig in SessionState #10050 (MichaelScofield)ceil, exp, factorial to datafusion-functions crate #10083 (erenavsarogullari)unnest multiple arrays #10044 (jonahgao)push_down_projections.rs to optimize_projections.rs #10071 (kavirajk)CountWildcardRule via TreeNode API #10066 (alamb)UnwrapCastInComparison to implement OptimizerRule::rewrite() #10087 (peter-toth)CREATE EXTERNAL TABLE #10080 (alamb)joins.slt #10120 (alamb)joins.slt #10122 (korowa)UnwrapCastInComparison to remove Expr clones #10115 (peter-toth)log::simplify and power::simplify #10086 (alamb)DataFrame supports unnesting multiple columns #10118 (jonahgao)Expr #10127 (peter-toth)37.1.0 changes to main #10136 (alamb)SetExpr during planning of SelectInto #10152 (jonahgao)NestedLoopJoinExec #9676 (korowa)TreeNode and LogicalPlan APIs to accept owned closures, deprecate transform_down_mut() and transform_up_mut() #10126 (peter-toth)TreeNode #10035 (alamb)Column::from(Tableref, &FieldRef), Expr::from(Column) and Expr::from(Tableref, &FieldRef) #10178 (alamb)Expr::is_volatile(), adjust TreeNode::exists() #10191 (peter-toth)TypeCoercion #10203 (alamb)Expr copies OptimizeProjection, 12% faster planning, encapsulate indicies #10216 (alamb)ParquetSink #10224 (wiedld)EliminateDuplicatedExpr optimizer pass to avoid clone #10218 (Lordworms)random() and uuid() produce unique values for each row #10248 (alamb)supports_zero_argument and avoid creating null array for empty args #10193 (jayzhan211)EliminateLimit #10253 (kevinmingtarja)substring function #10242 (jonahgao)ColumnarValue #10265 (alamb)IPCWriter metrics from u64 to usize #10278 (erenavsarogullari)array_has and get_field for Map #10148 (duongcongtoai)prefer_existing_union #10259 (NGA-TRAN)ScalarFunctionDefinition::Name #10277 (lewiszlw)preserve_partitioning on SortExec physical plan. #10153 (kavirajk)use ( " return internal_err!("UDF returned a different ...") #10317 (alamb)ExecutionPlan visitor example documentation #10286 (matthewmturner)RewriteDisjunctivePredicate #10305 (rohitrastogi)DecorrelatePredicateSubquery #10318 (alamb)PropagateEmptyRelation #10332 (dmitrybugakov)1.78 #10353 (alamb)Transformed::{update,map,transform})_data #10355 (alamb)SessionState and SessionContext #10350 (alamb)ORDER BY in window function definition #10370 (yyy1000)Covariance (Sample) covar / covar_samp to be a User Defined Aggregate Function #10372 (jayzhan211)coalesce, struct and named_strct expr_fn function to take multiple arguments #10321 (alamb)create_physical_expr to scalar_function #10387 (jayzhan211)SessionContext/SessionState::create_physical_expr() to create PhysicalExpressions from Exprs #10330 (alamb)