Skip to main content
Version: 0.9.0

Metrics

In this section, we will introduce the MetricsReporter and HoodieMetrics in Hudi. You can view the metrics-related configurations here.

MetricsReporter#

MetricsReporter provides APIs for reporting HoodieMetrics to user-specified backends. Currently, the implementations include InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.

JmxMetricsReporter#

JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.

Configurations#

The following is an example of JmxMetricsReporter. More detaile configurations can be referenced here.

hoodie.metrics.on=truehoodie.metrics.reporter.type=JMXhoodie.metrics.jmx.host=192.168.0.106hoodie.metrics.jmx.port=4001

Demo#

As configured above, JmxMetricsReporter will started JMX server on port 4001. We can start a jconsole to connect to 192.168.0.106:4001. Below is an illustration of monitoring Hudi JMX metrics through jconsole.

hudi_jxm_metrics.png

MetricsGraphiteReporter#

MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send HoodieMetrics to it.

Configurations#

The following is an example of MetricsGraphiteReporter. More detaile configurations can be referenced here.

hoodie.metrics.on=truehoodie.metrics.reporter.type=GRAPHITEhoodie.metrics.graphite.host=192.168.0.106hoodie.metrics.graphite.port=2003hoodie.metrics.graphite.metric.prefix=<your metrics prefix>

Demo#

As configured above, assuming a Graphite server is running on host 192.168.0.106 and port 2003, a running Hudi job will connect and report metrics data to it. Below is an illustration of monitoring hudi metrics through Graphite.

hudi_graphite_metrics.png

DatadogMetricsReporter#

DatadogMetricsReporter is an implementation of Datadog reporter. A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.

Configurations#

The following is an example of DatadogMetricsReporter. More detailed configurations can be referenced here.

hoodie.metrics.on=truehoodie.metrics.reporter.type=DATADOGhoodie.metrics.datadog.api.site=EU # or UShoodie.metrics.datadog.api.key=<your api key>hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
  • hoodie.metrics.datadog.api.site will set the Datadog API site, which determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
  • hoodie.metrics.datadog.api.key will set the api key.
  • hoodie.metrics.datadog.metric.prefix will help segregate metrics by setting different prefixes for different jobs. Note that it will use . to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.

Demo#

In this demo, we ran a HoodieDeltaStreamer job with HoodieMetrics turned on and other configurations set properly.

hudi_datadog_metrics.png

As shown above, we were able to collect Hudi's action-related metrics like

  • <prefix>.<table name>.commit.totalScanTime

  • <prefix>.<table name>.clean.duration

  • <prefix>.<table name>.index.lookup.duration

    as well as HoodieDeltaStreamer-specific metrics

  • <prefix>.<table name>.deltastreamer.duration

  • <prefix>.<table name>.deltastreamer.hiveSyncDuration

UserDefinedMetricsReporter#

Allows users to define a custom metrics reporter.

Configurations#

The following is an example of UserDefinedMetricsReporter. More detailed configurations can be referenced here.

hoodie.metrics.on=truehoodie.metrics.reporter.class=test.TestUserDefinedMetricsReporter

Demo#

In this simple demo, TestMetricsReporter will print all gauges every 10 seconds

public static class TestUserDefinedMetricsReporter     extends AbstractUserDefinedMetricsReporter {  private static final Logger log = LogManager.getLogger(DummyMetricsReporter.class);
  private ScheduledExecutorService exec = Executors.newScheduledThreadPool(1, r -> {      Thread t = Executors.defaultThreadFactory().newThread(r);      t.setDaemon(true);      return t;  });
  public TestUserDefinedMetricsReporter(Properties props, MetricRegistry registry) {    super(props, registry);  }
  @Override  public void start() {    exec.schedule(this::report, 10, TimeUnit.SECONDS);  }
  @Override  public void report() {    this.getRegistry().getGauges().forEach((key, value) ->       log.info("key: " + key + " value: " + value.getValue().toString()));  }
  @Override  public Closeable getReporter() {    return null;  }
  @Override  public void stop() {    exec.shutdown();  }}

HoodieMetrics#

Once the Hudi writer is configured with the right table and environment for HoodieMetrics, it produces the following HoodieMetrics, that aid in debugging hudi tables

  • Commit Duration - The amount of time it took to successfully commit a batch of records
  • Rollback Duration - Similarly, the amount of time taken to undo partial data left over by a failed commit (rollback happens automatically after a failing write)
  • File Level metrics - Shows the amount of new files added, versions, deleted (cleaned) in each commit
  • Record Level Metrics - Total records inserted/updated etc per commit
  • Partition Level metrics - number of partitions upserted (super useful to understand sudden spikes in commit duration)

These HoodieMetrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.

hudi_commit_duration.png