Skip to main content
Version: 0.7.0

GCS Filesystem

For Hudi storage on GCS, regional buckets provide an DFS API with strong consistency.

GCS Configs#

There are two configurations required for Hudi GCS compatibility:

  • Adding GCS Credentials for Hudi
  • Adding required jars to classpath

GCS Credentials#

Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the fs.defaultFS with your GCS bucket name and Hudi should be able to read/write from the bucket.

  <property>    <name>fs.defaultFS</name>    <value>gs://hudi-bucket</value>  </property>
  <property>    <name></name>    <value></value>    <description>The FileSystem for gs: (GCS) uris.</description>  </property>
  <property>    <name></name>    <value></value>    <description>The AbstractFileSystem for gs: (GCS) uris.</description>  </property>
  <property>    <name></name>    <value>GCS_PROJECT_ID</value>  </property>  <property>    <name></name>    <value>true</value>  </property>  <property>    <name></name>    <value>GCS_SERVICE_ACCOUNT_EMAIL</value>  </property>  <property>    <name></name>    <value>GCS_SERVICE_ACCOUNT_KEYFILE</value>  </property>

GCS Libs#

GCS hadoop libraries to add to our classpath