Roadmap
Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month! This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken down by areas on our stack.
Future Releases
Next major release : 0.13.0 (Jan 2023)
Release | Timeline |
---|---|
0.12.2 | Dec 2022 |
0.13.0 | Jan 2023 |
1.0.0 | TBD |
Transactional Database Layer
Feature | Target Release | Tracking |
---|---|---|
Metaserver for all metadata | 0.13.0 | Strawman design, RFC-36, HUDI-3345 |
Supervised table service planning and execution | 0.13.0 | RFC-43, HUDI-4147 |
Support of Change Data Capture (CDC) with Hudi change logs | 0.13.0 | RFC-51, HUDI-3478 |
Record-level index to speed up UUID-based upserts and deletes | 0.13.0 | RFC-08, HUDI-53 |
Consistent hashing index for dynamic buckets | 0.13.0 | RFC-42, HUDI-3000 |
Secondary index to improve query performance | 0.13.0 | RFC-52, HUDI-3907 |
Reducing write amplification with Log Compaction in MOR | 0.13.0 | RFC-48, HUDI-3580 |
Eager conflict detection for Optimistic Concurrency Control | 0.13.0 | RFC-56, HUDI-1575 |
Index Function for Optimizing Query Performance | 0.13.0 | RFC-63, HUDI-512 |
Lock-Free message queue to improve writing efficiency | 0.13.0 | RFC-53, HUDI-3963 |
Efficient bootstrap and migration of existing non-Hudi dataset | 1.0.0 onward | HUDI-1265 |
Lock free concurrency control | 1.0.0 onward | HUDI-3187 |
Non-blocking/Lock-free updates during clustering | 1.0.0 onward | HUDI-1042 |
Time Travel updates, deletes | 1.0.0 onward | |
General purpose support for multi-table transactions | 1.0.0 onward |
Programming APIs
Feature | Target Release | Tracking |
---|---|---|
Redesign and optimization of record payload abstraction | 0.13.0 | RFC-46, HUDI-3217 |
Optimized storage layout for cloud object stores | 0.13.0 | RFC-60, HUDI-3625 |
Support of verification with multiple event_time fields | 1.0.0 onward | RFC-59, HUDI-4569 |
Execution Engine Integration
Feature | Target Release | Tracking |
---|---|---|
Integrate column stats index with all query engines | 0.13.0 | RFC-58, HUDI-4552 |
Upgrade to Spark 3 as the default profile | 0.13.0 | HUDI-3431 |
Spark datasource V2 read | 1.0.0 onward | HUDI-4449 |
Materialized Views with incremental updates using Flink | 1.0.0 onward | |
SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) | 1.0.0 onward | |
Explore other execution engines/runtimes (Ray, native Rust, Python) | 1.0.0 onward |
Platform Services
Feature | Target Release | Tracking |
---|---|---|
Protobuf source for DeltaStreamer | 0.13.0 | RFC-57, HUDI-4399 |
Improving Hudi CLI features and usability | 0.13.0 | HUDI-1388 |
Hudi integration with Snowflake | 1.0.0 onward | RFC-41, HUDI-2832 |
Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others | 1.0.0 onward | HUDI-1896 |
Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) | 1.0.0 onward | Strawman design |