Skip to main content
Version: 0.9.0


In this section, we will introduce the MetricsReporter and HoodieMetrics in Hudi. You can view the metrics-related configurations here.


MetricsReporter provides APIs for reporting HoodieMetrics to user-specified backends. Currently, the implementations include InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.


JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.


The following is an example of JmxMetricsReporter. More detaile configurations can be referenced here.


As configured above, JmxMetricsReporter will started JMX server on port 4001. We can start a jconsole to connect to Below is an illustration of monitoring Hudi JMX metrics through jconsole.



MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send HoodieMetrics to it.


The following is an example of MetricsGraphiteReporter. More detaile configurations can be referenced here.<your metrics prefix>


As configured above, assuming a Graphite server is running on host and port 2003, a running Hudi job will connect and report metrics data to it. Below is an illustration of monitoring hudi metrics through Graphite.



DatadogMetricsReporter is an implementation of Datadog reporter. A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.


The following is an example of DatadogMetricsReporter. More detailed configurations can be referenced here. # or UShoodie.metrics.datadog.api.key=<your api key>hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
  • will set the Datadog API site, which determines whether the requests will be sent to (EU) or (US). Set this according to your Datadog account settings.
  • hoodie.metrics.datadog.api.key will set the api key.
  • hoodie.metrics.datadog.metric.prefix will help segregate metrics by setting different prefixes for different jobs. Note that it will use . to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.


In this demo, we ran a HoodieDeltaStreamer job with HoodieMetrics turned on and other configurations set properly.


As shown above, we were able to collect Hudi's action-related metrics like

  • <prefix>.<table name>.commit.totalScanTime

  • <prefix>.<table name>.clean.duration

  • <prefix>.<table name>.index.lookup.duration

    as well as HoodieDeltaStreamer-specific metrics

  • <prefix>.<table name>.deltastreamer.duration

  • <prefix>.<table name>.deltastreamer.hiveSyncDuration


Allows users to define a custom metrics reporter.


The following is an example of UserDefinedMetricsReporter. More detailed configurations can be referenced here.



In this simple demo, TestMetricsReporter will print all gauges every 10 seconds

public static class TestUserDefinedMetricsReporter     extends AbstractUserDefinedMetricsReporter {  private static final Logger log = LogManager.getLogger(DummyMetricsReporter.class);
  private ScheduledExecutorService exec = Executors.newScheduledThreadPool(1, r -> {      Thread t = Executors.defaultThreadFactory().newThread(r);      t.setDaemon(true);      return t;  });
  public TestUserDefinedMetricsReporter(Properties props, MetricRegistry registry) {    super(props, registry);  }
  @Override  public void start() {    exec.schedule(this::report, 10, TimeUnit.SECONDS);  }
  @Override  public void report() {    this.getRegistry().getGauges().forEach((key, value) ->"key: " + key + " value: " + value.getValue().toString()));  }
  @Override  public Closeable getReporter() {    return null;  }
  @Override  public void stop() {    exec.shutdown();  }}


Once the Hudi writer is configured with the right table and environment for HoodieMetrics, it produces the following HoodieMetrics, that aid in debugging hudi tables

  • Commit Duration - The amount of time it took to successfully commit a batch of records
  • Rollback Duration - Similarly, the amount of time taken to undo partial data left over by a failed commit (rollback happens automatically after a failing write)
  • File Level metrics - Shows the amount of new files added, versions, deleted (cleaned) in each commit
  • Record Level Metrics - Total records inserted/updated etc per commit
  • Partition Level metrics - number of partitions upserted (super useful to understand sudden spikes in commit duration)

These HoodieMetrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.