<!--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

Features

General

• SQL Parser
• SQL Query Planner
• DataFrame API
• Parallel query execution
• Streaming Execution

Optimizations

• Query Optimizer
• Constant folding
• Join Reordering
• Limit Pushdown
• Projection push down
• Predicate push down

SQL Support

• Type coercion
• Projection (SELECT)
• Filter (WHERE)
• Filter post-aggregate (HAVING)
• Sorting (ORDER BY)
• Limit (LIMIT)
• Aggregate (GROUP BY)
• cast /try_cast
• VALUES lists
• String Functions
• Conditional Functions
• Time and Date Functions
• Math Functions
• Aggregate Functions (SUM, MEDIAN, and many more)
• Schema Queries
  • SHOW TABLES
  • SHOW COLUMNS FROM <table/view>
  • SHOW CREATE TABLE <view>
  • Basic SQL Information Schema (TABLES, VIEWS, COLUMNS)
  • Full SQL Information Schema support
• Support for nested types (ARRAY/LIST and STRUCT.
  • Read support
  • Write support
  • Field access (col['field'] and [col[1]])
  • Array Functions
  • Struct Functions
    • struct
    • Postgres JSON operators (->, ->>, etc.)
• Subqueries
• Common Table Expressions (CTE)
• Set Operations (UNION [ALL], INTERSECT [ALL], EXCEPT[ALL])
• Joins (INNER, LEFT, RIGHT, FULL, CROSS)
• Window Functions
  • Empty (OVER())
  • Partitioning and ordering: (OVER(PARTITION BY <..> ORDER BY <..>))
  • Custom Window (ORDER BY time ROWS BETWEEN 2 PRECEDING AND 0 FOLLOWING))
  • User Defined Window and Aggregate Functions
• Catalogs
  • Schemas (CREATE / DROP SCHEMA)
  • Tables (CREATE / DROP TABLE, CREATE TABLE AS SELECT)
• Data Insert
  • INSERT INTO
  • COPY .. INTO ..
  • CSV
  • JSON
  • Parquet
  • Avro

Runtime

• Streaming Grouping
• Streaming Window Evaluation
• Memory limits enforced
• Spilling (to disk) Sort
• Spilling (to disk) Grouping
• Spilling (to disk) Sort Merge Join
• Spilling (to disk) Hash Join

Data Sources

In addition to allowing arbitrary datasources via the TableProvider trait, DataFusion includes built in support for the following formats:

• CSV
• Parquet
  • Primitive and Nested Types
  • Row Group and Data Page pruning on min/max statistics
  • Row Group pruning on Bloom Filters
  • Predicate push down (late materialization) not by default
• JSON
• Avro
• Arrow