Back to Hudi

RFCs

rfc/README.md

0.5.328.8 KB
Original Source
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

RFCs

The RFC process is documented on our site. Please familiarize yourself with it, before working a new RFC.

Status can be one of these values.

StatusMeaning
:eyes: UNDER REVIEWRFC has been proposed and community is actively debating the design/proposal.
:hammer_and_wrench: IN PROGRESSThe initial phase of implementation is underway.
:arrows_counterclockwise: ONGOINGSome or most work has landed; community continues to improve or build follow on phases.
:x: ABANDONEDThe proposal was not implemented, due to various reasons.
:white_check_mark: COMPLETEDAll work is deemed complete.

The list of all RFCs can be found here.

Older RFC content is still here.

RFC NumberTitleStatus
1CSV Source Support for Delta Streamer:white_check_mark: COMPLETED
2ORC Storage in Hudi:white_check_mark: COMPLETED
3Timeline Service with Incremental File System View Syncing:white_check_mark: COMPLETED
4Faster Hive incremental pull queries:white_check_mark: COMPLETED
5HUI (Hudi WebUI):x: ABANDONED
6Add indexing support to the log file:x: ABANDONED
7Point in time Time-Travel queries on Hudi table:white_check_mark: COMPLETED
8Metadata based Record Index:white_check_mark: COMPLETED
9Hudi Dataset Snapshot Exporter:white_check_mark: COMPLETED
10Restructuring and auto-generation of docs:white_check_mark: COMPLETED
11Refactor of the configuration framework of hudi project:x: ABANDONED
12Efficient Migration of Large Parquet Tables to Apache Hudi:white_check_mark: COMPLETED
13Integrate Hudi with Flink:white_check_mark: COMPLETED
14JDBC incremental puller:white_check_mark: COMPLETED
15HUDI File Listing Improvements:white_check_mark: COMPLETED
16Abstraction for HoodieInputFormat and RecordReader:white_check_mark: COMPLETED
17Abstract common meta sync module support multiple meta service:white_check_mark: COMPLETED
18Insert Overwrite API:white_check_mark: COMPLETED
19Clustering data for freshness and query performance:white_check_mark: COMPLETED
20handle failed records:arrows_counterclockwise: ONGOING
21Allow HoodieRecordKey to be Virtual:white_check_mark: COMPLETED
22Snapshot Isolation using Optimistic Concurrency Control for multi-writers:white_check_mark: COMPLETED
23Hudi Observability metrics collection:x: ABANDONED
24Hoodie Flink Writer Proposal:white_check_mark: COMPLETED
25Spark SQL Extension For Hudi:white_check_mark: COMPLETED
26Optimization For Hudi Table Query:white_check_mark: COMPLETED
27Data skipping index to improve query performance:white_check_mark: COMPLETED
28Support Z-order curve:white_check_mark: COMPLETED
29Hash Index:white_check_mark: COMPLETED
30Batch operation:x: ABANDONED
31Hive integration Improvement:x: ABANDONED
32Kafka Connect Sink for Hudi:arrows_counterclockwise: ONGOING
33Hudi supports more comprehensive Schema Evolution:white_check_mark: COMPLETED
34Hudi BigQuery Integration:white_check_mark: COMPLETED
35Make Flink MOR table writing streaming friendly:white_check_mark: COMPLETED
36HUDI Metastore Server:arrows_counterclockwise: ONGOING
37Hudi Metadata based Bloom Index:white_check_mark: COMPLETED
38Spark Datasource V2 Integration:white_check_mark: COMPLETED
39Incremental source for Debezium:white_check_mark: COMPLETED
40Connector for Trino:white_check_mark: COMPLETED
41Snowflake Integration, supported via Apache XTable (Incubating):x: ABANDONED
42Consistent Hashing Index:arrows_counterclockwise: ONGOING
43Table Management Service:x: ABANDONED
44Hudi Connector for Presto:white_check_mark: COMPLETED
45Asynchronous Metadata Indexing:white_check_mark: COMPLETED
46Optimizing Record Payload Handling:white_check_mark: COMPLETED
47Add Call Produce Command for Spark SQL:white_check_mark: COMPLETED
48LogCompaction for MOR tables:white_check_mark: COMPLETED
49Support sync with DataHub:white_check_mark: COMPLETED
50Improve Timeline Server:x: ABANDONED
51Change Data Capture:arrows_counterclockwise: ONGOING
52Introduce Secondary Index to Improve HUDI Query Performance:x: ABANDONED
53Use Lock-Free Message Queue Improving Hoodie Writing Efficiency:white_check_mark: COMPLETED
54New Table APIs and Streamline Hudi Configs:x: ABANDONED
55Improve Hive/Meta sync class design and hierarchies:white_check_mark: COMPLETED
56Early Conflict Detection For Multi-Writer:white_check_mark: COMPLETED
57DeltaStreamer Protobuf Support:white_check_mark: COMPLETED
58Integrate column stats index with all query engines:white_check_mark: COMPLETED
59Multiple event_time Fields Latest Verification in a Single Table:eyes: UNDER REVIEW
60Federated Storage Layer:eyes: UNDER REVIEW
61Snapshot view management:eyes: UNDER REVIEW
62Diagnostic Reporter:eyes: UNDER REVIEW
63Expression Indexes:arrows_counterclockwise: ONGOING
64New Hudi Table Spec API for Query Integrations:eyes: UNDER REVIEW
65Partition TTL Management:white_check_mark: COMPLETED
66Non Blocking Concurrency Control:white_check_mark: COMPLETED
67Hudi Bundle Standards:white_check_mark: COMPLETED
68A More Effective HoodieMergeHandler for COW Table with Parquet:x: ABANDONED
69Hudi 1.x:white_check_mark: COMPLETED
70Hudi Reverse Streamer:eyes: UNDER REVIEW
71Enhance OCC conflict detection:eyes: UNDER REVIEW
72Redesign Hudi-Spark Integration:arrows_counterclockwise: ONGOING
73Multi-Table Transactions:eyes: UNDER REVIEW
74HoodieStorage: Hudi Storage Abstraction and APIs:arrows_counterclockwise: ONGOING
75Hudi-Native HFile Reader and Writer:white_check_mark: COMPLETED
76Auto Record key generation:white_check_mark: COMPLETED
77Secondary Index:white_check_mark: COMPLETED
781.0 Migration:hammer_and_wrench: IN PROGRESS
79Robust handling of spark task retries and failures:x: ABANDONED
80Column Groups:hammer_and_wrench: IN PROGRESS
81Introduce Primary Key Sorted Table:eyes: UNDER REVIEW
82Concurrent schema evolution detection:white_check_mark: COMPLETED
83Incremental Table Service:white_check_mark: COMPLETED
84Optimized SerDe of DataStream in Flink operators:white_check_mark: COMPLETED
85Hudi Issue and Sprint Management in Jira:white_check_mark: COMPLETED
86DataFrame Implementation of HUDI write path:eyes: UNDER REVIEW
87Avro elimination for Flink writer:hammer_and_wrench: IN PROGRESS
88New Schema/DataType/Expression Abstractions:eyes: UNDER REVIEW
89Dynamic Partition Level Bucket Index:eyes: UNDER REVIEW
90Add support for cancellable clustering table service plans:eyes: UNDER REVIEW
91Storage-based lock provider using conditional writes:hammer_and_wrench: IN PROGRESS
92Support Bitmap Index:hammer_and_wrench: IN PROGRESS
93Pluggable Table Formats in Hudi:hammer_and_wrench: IN PROGRESS
94Hudi Timeline User Interface (UI):eyes: UNDER REVIEW
95Hudi Flink Source Based on FLIP-27:eyes: UNDER REVIEW
96Introduce Unified Bucket Index:eyes: UNDER REVIEW
97Deprecate Hudi Payload Class Usage:eyes: UNDER REVIEW
98Spark Datasource V2 Read:eyes: UNDER REVIEW
99Hudi Type System Redesign:eyes: UNDER REVIEW
100Unstructured Data Storage in Hudi:eyes: UNDER REVIEW
101Updates to the HoodieRecordMerger API:hammer_and_wrench: IN PROGRESS
102RLI support for Flink streaming:eyes: UNDER REVIEW
103Hudi LSM tree layout:eyes: UNDER REVIEW