Skip to main content
Version: Next

COS Filesystem

这个页面描述了如何让你的Hudi spark任务使用Tencent Cloud COS存储。

Tencent Cloud COS 部署​

为了让Hudi使用COS，需要增加两部分的配置:

  • 为Hidi增加Tencent Cloud COS的相关配置
  • 增加Jar包的MVN依赖

Tencent Cloud COS 相关的配置​

新增下面的配置到你的Hudi能访问的core-site.xml文件。使用你的COS bucket name替换掉fs.defaultFS，使用COS的key和secret分别替换fs.cosn.userinfo.secretKey和fs.cosn.userinfo.secretId。主要Hudi就能读写相应的bucket。

    <property>
        <name>fs.defaultFS</name>
        <value>cosn://bucketname</value>
        <description>COS bucket name</description>
    </property>

    <property>
        <name>fs.cosn.userinfo.secretId</name>
        <value>cos-secretId</value>
        <description>Tencent Cloud Secret Id</description>
    </property>

    <property>
        <name>fs.cosn.userinfo.secretKey</name>
        <value>cos-secretkey</value>
        <description>Tencent Cloud Secret Key</description>
    </property>

    <property>
        <name>fs.cosn.bucket.region</name>
        <value>ap-region</value>
        <description>The region where the bucket is located.</description>
    </property>

    <property>
        <name>fs.cosn.bucket.endpoint_suffix</name>
        <value>cos.endpoint.suffix</value>
        <description>
          COS endpoint to connect to.
          For public cloud users, it is recommended not to set this option, and only the correct area field is required.
        </description>
    </property>

    <property>
        <name>fs.cosn.impl</name>
        <value>org.apache.hadoop.fs.CosFileSystem</value>
        <description>The implementation class of the CosN Filesystem.</description>
    </property>

    <property>
        <name>fs.AbstractFileSystem.cosn.impl</name>
        <value>org.apache.hadoop.fs.CosN</value>
        <description>The implementation class of the CosN AbstractFileSystem.</description>
    </property>

Tencent Cloud COS Libs​

添加COS依赖jar包到classpath

  • org.apache.hadoop:hadoop-cos:2.8.5