Combining Spring Boot and Datadog StatsD Writer

Are you monitoring your Spring Boot app?

Spring Boot comes with the actuator package, which brings a lot of tools making application monitoring in production easier. It comes with an interesting metric export reader/writer feature, which help in collection of metrics and the publishing to a desired output channel. Out of the box Actuator supports writers to channels such as JMX, Redis or basically any output channel the developer wishes to integrate. In my case, I am using StatsD (with a local agent) reporting to Datadog. To define a StatsD writer, one can simply include a StatsD client dependency which will be picked up automatically by Spring Boot and defines a writer.

For those of you who don’t know, Datadog supports a powerful tagging feature for reporting. Tagging allows breaking a certain metric by dynamic values relevant to the metric. For example, a certain endpoint can return various status codes. Without the tagging feature, each status code of the endpoint must yield one metric. With the tagging feature one may just define the status code as a tag of a more general metric. This saves a lot of time when building graphs based on such metrics. Unfortunately, Spring boot does not support tagging as of yet, as the writer interface defined in MetricWriter does not accept tags as arguments. That’s a shame, since this means that either we must give up the tagging feature and enjoy the auto-magic of Actuator, or we have to implement the reporting ourselves. Turns out, the latter isn’t so difficult, thanks to Coursera’s Dropwizard Datadog metrics integration library. This functionality be achieved in three simple steps.

First, we create a UDP Transport object that bridges the application with the local StatsD agent.

@Bean
public UdpTransport udpTransport(StatsDConfiguration configuration) {
    return new UdpTransport.Builder()
            .withPort(configuration.getPort())
            .withPrefix(configuration.getPrefix())
            .withStatsdHost(configuration.getHost())
            .build();
}

Actuator is already including Dropwizard metric library, so actually we can autowire and leverage codehale’s MetricRegistry bean, and supply it as the registry for the DatadogReporter.

final DatadogReporter reporter = DatadogReporter
        .forRegistry(registry)
        .withTransport(udpTransport)
        .convertDurationsTo(TimeUnit.MILLISECONDS)
        .build();
reporter.start(60,TimeUnit.SECONDS);

Now that we have our report all we need to do is define a process of preparing reporting. This can be done in two ways: by calling the UDP Transport directly (low level) or by registering metrics with MetricRegistry (high level), in which case, the reporter will take care for publishing it.
In the following example, I am using a Spring Boot Actuator MetricReaderPublicMetrics which holds information about endpoint interactions with the server. The goal is to supplying endpoint path and status code as tags to datadog. Additionally, I am also reporting on timing of endpoints by supplying the endpoint path as a tag.

Example 1: Reporting directly via supplied UDPTransport

    @Scheduled(fixedDelay = 30000, initialDelay = 30000)
    void exportPublicMetricsAndTransport() throws IOException {
        for (Metric<?> metric : metricReaderPublicMetrics.metrics()) {
            if (metric.getName().startsWith("counter.status.")) {
                final String statusCode = metric.getName().substring(15, 18);
                final String endpoint = metric.getName().substring(19);
                final List<String> tags = Arrays.asList("endpoint:" + endpoint, "status-code:" + statusCode);
                transport.prepare().addGauge(new DatadogGauge("endpoint.counter",
                        metric.getValue().longValue(),
                        metric.getTimestamp().getTime() / 1000,
                        HOSTNAME,
                        tags));
            }
            else if (metric.getName().startsWith("gauge.response.")) {
                transport.prepare()
                        .addGauge(new DatadogGauge("endpoint.timer",
                                metric.getValue().longValue(),
                                metric.getTimestamp().getTime() / 1000,
                                HOSTNAME,
                                Collections.singletonList(metric.getName().substring(15))));
            }
        }
    }

Example 2: Reporting by registering gauges via supplied MetricRegistry

    @Scheduled(fixedDelay = 30000, initialDelay = 30000)
    void exportPublicMetricsAndRegister() throws IOException {
        for (Metric<?> metric : metricReaderPublicMetrics.metrics()) {

            final Gauge<Long> gauge = () -> metric.getValue().longValue();
            if (metric.getName().startsWith("counter.status.")) {

                final String statusCode = metric.getName().substring(15, 18);
                final String endpoint = metric.getName().substring(19);

                final String countMetricWithTags = "endpoint.counter[" +
                        String.join(",", "endpoint:" + endpoint, "status-code:" + statusCode)
                        + "]";
                metricRegistry.remove(countMetricWithTags); // to avoid IllegalArgumentException
                metricRegistry.register(countMetricWithTags, gauge);

            } else if (metric.getName().startsWith("gauge.response.")) {
                final String timerMetricWithTag = "endpoint.timer[endpoint:" + metric.getName().substring(15) + "]";
                metricRegistry.remove(timerMetricWithTag);
                metricRegistry.register(timerMetricWithTag, gauge);
            }
        }
    }

Note the format of the metric name: “metric[tag_1_key:tag_1:value,…]”, which is important, as the reporter would later extract the tags out of it.

To test that the reporting works correctly, we’ll listen on port 8125:

nc -u -l 8125
pepapi.endpoint.counter:4|g|#status-code:200,endpoint:api.v1.games

And there it is! We are reporting to Datadog from our Spring Boot app. As a matter of fact, this solution could be applied to any app.

The full code example can be viewed here.