Skip to main content

Monitor Hudi metrics with Datadog

rxu
2 min read
Tags:

Availability​

0.6.0 (unreleased)

Introduction​

Datadog is a popular monitoring service. In the upcoming 0.6.0 release of Apache Hudi, we will introduce the feature of reporting Hudi metrics via Datadog HTTP API, in addition to the current reporter types: Graphite and JMX.

Configurations​

Similar to other supported reporters, turning on Datadog reporter requires these 2 properties.

hoodie.metrics.on=true
hoodie.metrics.reporter.type=DATADOG

The following property sets the Datadog API site. It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.

hoodie.metrics.datadog.api.site=EU # or US

The property hoodie.metrics.datadog.api.key allows you to set the api key directly from the configuration.

hoodie.metrics.datadog.api.key=<your api key>
hoodie.metrics.datadog.api.key.supplier=<your api key supplier>

Due to security consideration in some cases, you may prefer to return the api key at runtime. To go with this approach, implement java.util.function.Supplier<String> and set the implementation's FQCN to hoodie.metrics.datadog.api.key.supplier, and make sure hoodie.metrics.datadog.api.key is not set since it will take higher precedence.

The following property helps segregate metrics by setting different prefixes for different jobs.

hoodie.metrics.datadog.metric.prefix=<your metrics prefix>

Note that it will use . to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.

There are other optional properties, which are explained in the configuration reference page.

Demo​

In this demo, we ran a HoodieDeltaStreamer job with metrics turn on and other configurations set properly.

datadog metrics demo

As shown above, we were able to collect Hudi's action-related metrics like

  • <prefix>.<table name>.commit.totalScanTime
  • <prefix>.<table name>.clean.duration
  • <prefix>.<table name>.index.lookup.duration

as well as HoodieDeltaStreamer-specific metrics

  • <prefix>.<table name>.deltastreamer.duration
  • <prefix>.<table name>.deltastreamer.hiveSyncDuration