ContextQMD
Back to Datafusion

14.0.0

dev/changelog/14.0.0.md

53.1.043.5 KB
Original Source
<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

14.0.0 (2022-11-04)

Full Changelog

Breaking changes:

  • Improve FieldNotFound errors #4084 [sql] (andygrove)
  • Refactor: move simplify_expression.rs and expr_simplifier.rs to a new mod simplify_expressions #3951 (HaoYang670)
  • Support for non-u64 types for Window Bound #3916 [sql] (mustafasrepo)
  • Expose parquet reader settings using normal DataFusion ConfigOptions #3822 (alamb)
  • Add Filter::try_new with validation #3796 [sql] (andygrove)
  • Change public simplify API and add a public coerce API #3758 (alamb)

Implemented enhancements:

  • Automatically register tables if ObjectStore root is configured #4094
  • Simplify small InList expressions #4089
  • Support SET command #4067
  • add uuid() function to generate unique uuid per row #4045
  • Publish benchmark crate so that it can be used as a library in Ballista #4016
  • Add statistics methods to TableProvider trait for use in cost-based optimizations in the logical plan #3983
  • Implement current_time Function #3982
  • Implement current_date Function #3981
  • Put common code used for testing code into datafusion/test_utils.rs #3960
  • Print the configurations of ConfigOptions in an ordered way so that we can directly compare the equality of two ConfigOptions by their debug strings #3952
  • Don't make dependants install protoc #3947
  • Implement right anti join and support it in HashBuildProbeOrder #3946
  • Implement right semi join and support it in HashBuildProbeOrder #3945
  • Refactor simplify_expressions and expr_simplifier #3934
  • Implement serialization for ScalarValue::FixedSizeBinary #3928
  • Support inlining view / dataframes logical plan #3913
  • Plans with tables from TableProviderFactorys can't be serialized #3906
  • Simplify a AND a and a OR a. #3895
  • Allow configuring statistics on TPC-H benchmarks #3888
  • CI checks stuck in queued mode #3883
  • Multiple optimizer passes #3879
  • datafusion-proto does not support view table scan #3874
  • TableProviderFactories need to be async and return a Result to be useful #3866
  • Factorize common AND factors out of OR predicates to support filterPushDown as possible #3858
  • Replace concat_ws with concat when the delimiter is empty string #3857
  • Concatenate contiguous literal arguments of concat_ws when doing the expression simplification #3856
  • Partition and Sort Enforcement #3854
  • Enable mimalloc by default in benchmarks #3851
  • Add collect statistics configuration #3847
  • [SQL] - Support cache/uncache table syntax #3842
  • Filter pushdown doesn't seem to apply for filter on TPC-H Q17 #3839
  • Support pushdown multi-columns in PageIndex pruning. #3834
  • Consolidate Expr manipulation code so it is more discoverable and make it easier to use #3808
  • Leverage input array's null buffer for regex replace to optimize sparse arrays #3803
  • Improve join cardinality estimation when there is no overlap in the min/max values #3802
  • datafusion-cli up to date check is failing on master #3798
  • Optimize benchmark q2 subquery filter #3789
  • Benchmark should infer schema when running against Parquet #3776
  • Allow specialized physical functions to provide hints for the array adapter #3762
  • [User Guide] Add EXPLAIN to SQL reference #3755
  • move type coercion for agg/agg udf #3752
  • Prevent Cargo.lock for datafusion-cli being out-of-date #3744
  • Add example of expr apis including simplification and coercion #3740
  • support type coercion for ScalarFunction expr in the logical phase #3731
  • Add support for DISTINCT projections in decorrelate_where_exists #3724
  • Add type coercion rule for CONCAT and CONCAT_WS #3720
  • Expose and document a simpler public API for simplify expressions #3709
  • Expose + document the type coercion API publicly #3708
  • Concatenate contiguous literal arguments of CONCAT during the expression simplification. #3683
  • DataFusion 13.0.0 Release #3671
  • Add division by 0 rules in the expression simplification #3663
  • Compressed CSV/JSON Read #3641
  • remove type coercion for agg #3623
  • extract or clause as predicate for join rels #3577
  • Improve performance of regex_replace #3518
  • Add benchmarks for parquet queries with filter pushdown enabled #3457
  • Make type coercion rule more robust #3390
  • ViewTable::scan ignores filters and limits #3249
  • Add CREATE VIEW documentation to user guide #3211
  • Push additional parquet filtering into the parquet scan [EPIC] #3147
  • Remove core/logical_plan module #2683
  • Datafusion Optimizer Enhancement #2255
  • [Optimizer] Eliminate self compare self #2252
  • Break datafusion crate into smaller crates #1750
  • Benchmark constellation-rs/amadeus's parquet implementation #1341
  • Use parquet2 async reader in physical_plan/parquet #1058
  • Table Scan Enhancement Plan #944
  • Implement parquet page-level skipping with column index, using min/max stats #847
  • Support min/max statistics in ParquetTable and ParquetExec #537

Fixed bugs:

  • Clippy failing on master #4100
  • Panic when the number of partitions of the pipeline that throws the exception is inconsistent with the number of partitions output by the query #4096
  • FieldNotFound when field is available #4083
  • SingleDistinctToGroupBy being applied too broadly #4082
  • single_distinct_to_groupby strips qualifiers from group-by expressions #4049
  • Another Internal error when parquet predicate pushdown is enabled "Error evaluating filter predicate: #4046
  • Decimal multiplied by Float produces incorrect results #4035
  • Cannot query external table - TableScan replaced with EmptyExec #4027
  • benchmark q17 produces incorrect result #4026
  • benchmark q14 produces incorrect result #4025
  • benchmark q11 producing incorrect results #4023
  • Internal error when parquet predicate pushdown is enabled "Error evaluating filter predicate:" #4006
  • Incorrect results with parquet filtering pushdown enabled #4005
  • Wrong results when parquet page index filtering is enabled #4002
  • Output schema of semi join has invalid projection added after HashBuildProbeOrder #4001
  • async deserialization functions are unintuitive and possibly insecure #3977
  • Expr::to_bytes can produce output that hits Expr::from_bytes recursion limit #3968
  • Bug on propagating arrow field metadata #3964
  • Predicate still has cast when comparing Timestamp(Nano, None) to a timestamp literal, so can't be pushed down or used for pruning #3938
  • Error using IN list on dictionary encoded data: InList does not support datatype Dictionary(Int32, Utf8). #3936
  • Internal error in CAST from Timestamp[us] #3922
  • ScalarValue not implemented for FixedSizeBinary types #3910
  • [DOC] - There are unsupported DDL in the official documentation #3904
  • datafusion-proto deserialize with Substring(str [from int] [for int]) fails #3901
  • count(Literal) gives wrong column name #3891
  • projection_push_down adds duplicate projections with multiple passes #3881
  • Default physical planner generates empty relation for DROP TABLE, CREATE MEMORY TABLE, etc #3873
  • Binary expression canonical names are incorrect in some cases #3865
  • Using the window function lag causes panic. #3830
  • chrono crate : specify 0.4.22 as the minimum version due to spurious build failures #3827
  • datafusion-proto deserialize with q16 sql fails #3820
  • Filter predicates should not be aliased #3795
  • Write csv not save all lines of dataframe #3783
  • Regression in simplifying expressions in subqueries #3760
  • DataFusionError(Internal("The size of the sorted batch is larger than the size of the input batch: 2120 > 2312")) #3747
  • "labeler" PR check is broken #3743
  • DataFrame::select_columns doesn't work with names containing "." #3733
  • TPC-H Query 1 has regressed #3729
  • [RUST][Datafusion] What causes "Error: Execution("file size of 4 is less than footer")" error? #3800
  • Field names containing periods such as f.c cannot work #3682
  • TableProvider implementation for DataFrame does not support filter pushdown #3681
  • using Decimal(0) make system panicked #3665
  • Cannot query some parquet files in S3, but they work locally #3633
  • col / col returns 1 when col = 0 #3615
  • register_csv allow space in table_path #3589
  • Hardcoded u64 for WindowFrameBound fields #3571
  • docs.rs cannot build datafusion-proto crate #3538
  • Row Hash loads whole aggregation state to memory before sending #3460
  • approx_percentile_cont return wrong result when scan multi parquet files. #3140
  • User guide is incorrect regarding using CLI to register CSV files using schema inference #3001
  • Exception: Internal error, Exception: Schema error #2938
  • Version 0.6.0 Panic error during SQL execution #2738
  • wrong result when operation parquet #2044
  • Local object store accepts file:/// as base path, but LocalStore returns meta without the prefix. #1923
  • Reading nested parquet files results in index out of bounds #1383
  • - (negation) with NULL literals does not work: can't be evaluated because the expression's type is Utf8, not signed #1192
  • Inconsistent cast behavior #957
  • single_distinct_to_groupby no longer drops qualifiers #4050 [sql] (andygrove)

Documentation updates:

  • Clarify in docs that Identifiers are made lower-case in SQL query #2374
  • Fix broken links in contributor guide #3956 (Jefffrey)
  • add create view explanation #3925 (retikulum)
  • Update datafusion-examples README #3814 (alamb)
  • Add Seafowl to list of projects using DataFusion #3792 (mildbyte)

Closed issues:

  • [QUESTION] How many times should be the function create_name called when executing a query? #3900
  • Improve the Expr string format #3878
  • Simplify division by zero (division by one / multiplication by zero / multiplication by one) for Decimal types as well #3643
  • InList: merge check branch #2833
  • Optimization InList: compare the float data type using OrderedFloat<T> #2831
  • Outdated section of the add function of the contribution guide #2560
  • Optimize InList implementation with native types rather than ScalarValue #2165
  • Improve testing of optimizers using EXPLAIN #1118
  • Crash on parsing sql query with Cyrillic letters #184
  • [EPIC] Support all TPC-H queries in benchmark #158
  • Implement optional second argument to ltrim and rtrim functions #144
  • Benchmark crate does not have a SIMD feature #124
  • ColumnarValue::into_array should not require batch #113
  • [Rust] Parquet data source does not support complex types #83

Merged pull requests:

  • Appease new clippy #4101 (alamb)
  • minor: Split parquet reader up into smaller modules #4099 (alamb)
  • [MINOR] Update SET in cli.md #4098 (waitingkuo)
  • fix: Scheduler panic routing errors #4097 (yukkit)
  • Automatically register tables if ObjectStore root is configured #4095 (avantgardnerio)
  • minor: Use Operator::swap #4092 (alamb)
  • Simplify small InListExpr #4090 (Dandandan)
  • Minor: Add arrow-rs ticket reference and turn some comments into docstrings #4088 (alamb)
  • Support Dictionary in InListExpr #4070 (tustvold)
  • support SET variable #4069 [sql] (waitingkuo)
  • Add in list bench #4068 (tustvold)
  • Improve Error Handling and Readibility for downcasting StructArray #4061 (retikulum)
  • Build tests separately from running #4060 (alamb)
  • Simplify InListExpr ~20-70% Faster #4057 (tustvold)
  • MINOR: Print unoptimized logical plan in execute_query of tpch benchmark #4056 (viirya)
  • Minor: clean the code in eliminate_filter #4055 (HaoYang670)
  • Implement current_time scalar function #4054 (naosense)
  • Cleanup hash_utils adding support for decimal256 and f16 #4053 (tustvold)
  • Fix multicolumn parquet predicate pushdown (#4046) #4048 (tustvold)
  • Add CI checks that we can serde all benchmark queries #4047 (andygrove)
  • Enable more benchmark verification tests #4044 (andygrove)
  • Extract common parquet testing code to parquet-test-util crate #4042 (alamb)
  • add uuid() function #4041 (Jimexist)
  • Update to arrow 26, change timezones #4039 [sql] (tustvold)
  • Fix Decimal and Floating type coerce rule #4038 (viirya)
  • Reserve the literal expression of Count function #4031 [sql] (HaoYang670)
  • Implement current_date scalar function #4022 (comphead)
  • Fix predicate pushdown bugs: project columns within DatafusionArrowPredicate (#4005) (#4006) #4021 (tustvold)
  • minor: remove redundant code/TODO #4019 (jackwener)
  • Add CI check to verify that benchmark queries return the expected results #4015 (andygrove)
  • Minor: Add TODO and tracking ticket reference #4012 (alamb)
  • Add right anti join support and support it in HashBuildProbeOrder #4011 (Dandandan)
  • MINOR: Generate expected benchmark query results #4010 (andygrove)
  • Minor: remove unecessary clippy allow #4008 (alamb)
  • Minor: Do what clippy says and clean up some code #4007 (alamb)
  • Improve Error Handling and Readibility for downcasting Date32Array #4004 (retikulum)
  • Don't add projection for semi joins in HashBuildProbeOrder #4000 (Dandandan)
  • Minor: use DataType::is_nested #3995 (alamb)
  • [minor] bump prettier version #3992 (Jimexist)
  • Add parquet predicate pushdown metrics #3989 (alamb)
  • Pin datafusion-proto build dependencies #3987 (tustvold)
  • Add TableProvider.statistics method #3986 (andygrove)
  • Add Pull Request guidelines to contributor guide #3985 (alamb)
  • Update protos #3979 (tustvold)
  • Revert async changes but keep deltalake working #3978 (avantgardnerio)
  • Correctness integration test for parquet filter pushdown #3976 (alamb)
  • MINOR: Stop pretty printing batches in benchmark when there are no results #3974 (andygrove)
  • MINOR: Re-export Cast struct #3971 (andygrove)
  • fix: check recursion limit in Expr::to_bytes #3970 (crepererum)
  • [Part1] Partition and Sort Enforcement, PhysicalExpr enhancement #3969 (mingmwang)
  • Support pushdown multi-columns in PageIndex pruning. #3967 (Ted-Jiang)
  • Fix benchmarks README formatting #3966 (Jefffrey)
  • Bug fix on DFField to Field conversion: preserve metadata #3965 (metesynnada)
  • Informative Error Message for LAG and LEAD functions #3963 (mustafasrepo)
  • Minor: Add some docstrings to FileScanConfig and RuntimeEnv #3962 (alamb)
  • Move common code used for testing code into datafusion/test_utils #3961 (alamb)
  • Update minimum chrono dependency to 0.4.22 #3959 (alamb)
  • Implement right semi join and support in HashBuildProbeorder #3958 (Dandandan)
  • Print the configurations of ConfigOptions in an ordered way so that we can directly compare the equality of two ConfigOptions by their debug strings #3953 (yahoNanJing)
  • Vendor Generated Protobuf Code (#3947) #3950 (tustvold)
  • Implement serialization for ScalarValue::FixedSizeBinary #3943 (retikulum)
  • Consolidate physical join code into datafusion/core/src/physical_plan/joins #3942 (alamb)
  • Add optimizer test for simplifying predicates on timestamps #3939 (alamb)
  • Add test for querying predicate on dictionary #3937 (alamb)
  • fix: return error for unsupported SQL #3933 (Kikkon)
  • doc: fix doc about CREATE TABLE IF NOT EXISTS #3932 (jackwener)
  • Refactor Expr::Cast to use a struct. #3931 [sql] (jackwener)
  • minor: fix some typo. #3930 (jackwener)
  • chore: update cranelift-related dependencies #3926 (xudong963)
  • Change cast error from Internal to NotImplemented #3924 (alamb)
  • Support inlining view / dataframes logical plan #3923 (Dandandan)
  • Add test for Simplify redundant predicates #3915 (src255)
  • Implement ScalarValue for FixedSizeBinary #3911 (maxburke)
  • Add serde for plans with tables from TableProviderFactorys #3907 (avantgardnerio)
  • Support filter/limit pushdown for views/dataframes #3905 (Dandandan)
  • Factorize common AND factors out of OR predicates to support filterPu… #3903 (Ted-Jiang)
  • Add Substring(str [from int] [for int]) support in datafusion-proto #3902 (r4ntix)
  • Revert "Factorize common AND factors out of OR predicates to supportfilter Pu… (#3859)" #3897 (alamb)
  • MINOR: Add notes on Apache Reporter #3893 (andygrove)
  • Allow configuring collection of statistics during TPC-H benchmarks #3889 (isidentical)
  • Improve formatting of binary expressions #3884 [sql] (andygrove)
  • Multiple optimizer passes #3880 (andygrove)
  • [MINOR] Update docs with newly added configuration values #3877 (alamb)
  • [MINOR] Add a hint about how to resolve the Cargo.lock CI check #3876 (alamb)
  • Add LogicalPlan::ViewTable support in datafusion-proto #3875 (r4ntix)
  • Optimize the concat_ws function #3869 (HaoYang670)
  • Implement foundational filter selectivity analysis #3868 (isidentical)
  • Update TableProviderFactory trait to support real-world use-cases #3867 (avantgardnerio)
  • put subquery's equal clause into join on clauses instead of filter cl… #3862 (AssHero)
  • Factorize common AND factors out of OR predicates to support filterPu… #3859 (Ted-Jiang)
  • Enable mimalloc by default in benchmark #3853 (Dandandan)
  • Refactor Expr::Between to use a struct #3850 [sql] (b41sh)
  • Handle cardinality estimation for disjoint inner and outer joins #3848 (isidentical)
  • Add setting for statistics collection #3846 (Dandandan)
  • Update to arrow 25.0.0 #3844 [sql] (tustvold)
  • Tweak list of optimization rules #3841 (Dandandan)
  • Refactor Expr::GetIndexedField to use a struct #3838 [sql] (ygf11)
  • Infer the count of maximum distinct values from min/max #3837 (isidentical)
  • Refactor Expr::Like, Expr::ILike, Expr::SimilarTo to use a struct #3836 [sql] (b41sh)
  • Refactor Expr::BinaryExpr to use a struct #3835 [sql] (zhoudongyan)
  • update postgres version to 15 in integration test #3831 (Jimexist)
  • Fix the panic when lpad/rpad parameter is negative #3829 (ZuoTiJia)
  • MINOR: Document SHOW ALL in the users guide #3826 (alamb)
  • MINOR: Add datafusion-cli documentation on showing configuration #3825 (alamb)
  • Add/Remove Division Rules #3824 (retikulum)
  • Minor: Sort the output of SHOW ALL by config name #3823 [sql] (alamb)
  • Add precision != 0 check when making decimal type #3818 [sql] (HaoYang670)
  • Infer schema when running benchmarks against parquet #3817 (andygrove)
  • Finish removing deprecated datafusion::logical_plan module #3816 (andygrove)
  • Clarify initial example with respect to capitalization #3815 (alamb)
  • Improve expression simplification by running it twice #3811 (alamb)
  • Make expression manipulation consistent and easier to use: combine/split filter conjunction, etc #3810 (alamb)
  • Consolidate expression manipulation functions into datafusion_optimizer #3809 (alamb)
  • Optimize regexp_replace when the input is a sparse array #3804 (isidentical)
  • Stop ignoring errors when writing DataFrame to csv, parquet, json #3801 (andygrove)
  • Update datafusion-cli Cargo.lock to fix CI check on master #3799 (alamb)
  • MINOR: Benchmark regression tests #3790 (andygrove)
  • MINOR: Optimizer example and docs, deprecate Expr::name #3788 (andygrove)
  • Join cardinality computation for cost-based nested join optimizations #3787 (isidentical)
  • Optimizer now simplifies multiplication, division, module arg is a literal Decimal zero or one #3782 (drrtuy)
  • Implement parquet page-level skipping with column index, using min/ma… #3780 (Ted-Jiang)
  • Bump actions/labeler from 4.0.1 to 4.0.2 #3779 (dependabot[bot])
  • MINOR: correct ListingOptions.try_new docs to include the enabled stat collection #3775 (isidentical)
  • Teach a negative NULL expression to return NULL instead of an error #3771 (drrtuy)
  • Add benchmarks for testing row filtering #3769 (thinkharderdev)
  • move type coercion of agg and agg_udaf to logical phase #3768 (liukun4515)
  • User Guide: Add EXPLAIN to SQL reference #3767 (unvalley)
  • Allow specialized implementations to produce hints for the array adapter #3765 (isidentical)
  • Fix optimizer regression with simplifying expressions in subquery filters #3764 (andygrove)
  • Run all datafusion-examples in CI tests #3761 (alamb)
  • MINOR: Remove deprecated module datafusion::logical_plan::plan #3759 (andygrove)
  • Refactor Expr::Case to use a struct #3757 [sql] (andygrove)
  • Do not run labeler CI check if it would fail due to permissions #3756 (alamb)
  • MINOR: Improvements to scalar_subquery_to_join error handling #3754 (andygrove)
  • Always track the final size of the in-mem sorted arrays #3753 (isidentical)
  • Fix DataFrame::select_columns to handle column names with a period #3751 (zhoudongyan)
  • Fix ListingTableUrl to decode percent #3750 (unvalley)
  • remove type coercion for physical ScalarFunction #3749 (liukun4515)
  • CI: Add a new run to check whether datafusion-cli lock file is up-to-date #3745 (isidentical)
  • Add datafusion example of expression apis #3741 (alamb)
  • fix subquery where exists distinct #3732 (b41sh)
  • Remove some uneeded code in CommonSubexprEliminate #3730 (alamb)
  • Consolidate and better tests for expression re-rewriting / aliasing #3727 (alamb)
  • Fix output schema generated by CommonSubExprEliminate #3726 (alex-natzka)
  • Add type coercion rule for concat and concat_ws #3721 (HaoYang670)
  • Expose and document a simpler public API for simplify expressions #3719 (ygf11)
  • Remove dead code in UnwrapCastExprRewriter that may mask errors #3703 (alamb)
  • Fix DataFrame::with_column to handle creating column names with a period #3700 (alamb)
  • Add simplification rules for the CONCAT function #3684 (HaoYang670)
  • Compressed CSV/JSON support #3642 [sql] (Licht-T)
  • Simplify serialization by removing redundant PrimitiveScalarValue #3612 (alamb)
  • Pushdown single column predicates from ON join clauses #3578 (AssHero)
  • Simplify the serialization of ScalarValue::List #3547 (alamb)
  • Generate hash aggregation output in smaller record batches #3461 (milenkovicm)
  • Improve doc on lowercase treatment of columns on SQL #3385 (nanicpc)