Skip to main content
Version: Current

Streaming Writes

Spark Streaming​

You can write Hudi tables using spark's structured streaming.

# pyspark
# prepare to stream write to new table
streamingTableName = "hudi_trips_cow_streaming"
baseStreamingPath = "file:///tmp/hudi_trips_cow_streaming"
checkpointLocation = "file:///tmp/checkpoints/hudi_trips_cow_streaming"

hudi_streaming_options = {
'hoodie.table.name': streamingTableName,
'hoodie.datasource.write.recordkey.field': 'uuid',
'hoodie.datasource.write.partitionpath.field': 'partitionpath',
'hoodie.datasource.write.table.name': streamingTableName,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.table.ordering.fields': 'ts',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2
}

# create streaming df
df = spark.readStream
.format("hudi")
.load(basePath)

# write stream to new hudi table
df.writeStream.format("hudi")
.options(**hudi_streaming_options)
.outputMode("append")
.option("path", baseStreamingPath)
.option("checkpointLocation", checkpointLocation)
.trigger(once=True)
.start()

Session settings and upgrade compatibility​

Starting with 1.3.0, Structured Streaming sinks honor the active session's spark.hoodie.* settings. Previously, the streaming sink ignored these settings, even though batch DataFrame writes supported them. Review ambient session configuration when upgrading an existing writeStream application.

For example, this setting selects inserts for a subsequently started streaming query:

spark.conf.set("spark.hoodie.datasource.write.operation", "insert")

Explicit writer options take precedence. To retain upsert behavior despite that session setting:

query = (df.writeStream.format("hudi")
.options(**hudi_streaming_options)
.option("hoodie.datasource.write.operation", "upsert")
.option("path", baseStreamingPath)
.option("checkpointLocation", checkpointLocation)
.start())

The precedence is explicit writer/storage options, then session spark.hoodie.*, then Hudi defaults. The sink strips the spark. prefix before validation and writing. Explicit options may use either prefix; if both forms are supplied explicitly, the canonical hoodie.* option wins. Bare hoodie.* session settings are not forwarded.

Settings are captured when the sink is created. Changing the session afterward does not change an existing query; restart it to capture new settings. The same captured options control multi-writer validation, checkpoint deduplication, and micro-batch writes. Table identity options must still agree with the persisted Hudi table. An incompatible record key, ordering field, partition field, key generator, or record-merge setting fails validation when opening an existing table, including during checkpoint replay.

Blogs