DataHub
DataHub is a rich metadata platform that supports features like data discovery, data obeservability, federated governance, etc.
Since Hudi 0.11.0, you can now sync to a DataHub instance by setting DataHubSyncTool as one of the sync tool classes
for HoodieStreamer.
The target Hudi table will be sync'ed to DataHub as a Dataset. The Hudi table's avro schema will be sync'ed, along
with the commit timestamp when running the sync.
Configurations
DataHubSyncTool makes use of DataHub's Java Emitter to send the metadata via HTTP REST APIs. It is required to
set hoodie.meta.sync.datahub.emitter.server to the URL of the DataHub instance for sync.
If needs auth token, set hoodie.meta.sync.datahub.emitter.token.
If needs customized creation of the emitter object,
implement org.apache.hudi.sync.datahub.config.DataHubEmitterSupplier and supply the implementation's FQCN
to hoodie.meta.sync.datahub.emitter.supplier.class.
By default, the sync config's database name and table name will be used to make the target Dataset's URN.
Subclass HoodieDataHubDatasetIdentifier and set it to hoodie.meta.sync.datahub.dataset.identifier.class to customize
the URN creation.
Example
The following shows an example configuration to run HoodieStreamer with DataHubSyncTool.
In addition to hudi-utilities-slim-bundle that contains HoodieStreamer, you also add hudi-datahub-sync-bundle to
the classpath.
spark-submit --master yarn \
--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
--jars /opt/hudi-datahub-sync-bundle-1.0.0.jar \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
/opt/hudi-utilities-slim-bundle_2.12-1.0.0.jar \
--target-table mytable \
# ... other HoodieStreamer's configs
--enable-sync \
--sync-tool-classes org.apache.hudi.sync.datahub.DataHubSyncTool \
--hoodie-conf hoodie.meta.sync.datahub.emitter.server=http://url-to-datahub-instance:8080 \
--hoodie-conf hoodie.datasource.hive_sync.database=mydb \
--hoodie-conf hoodie.datasource.hive_sync.table=mytable \