Skip to main content
Skip table of contents

StatsD Metrics

Introduction

This document is meant to provide an introduction to how StatsD metrics are sent and how a developer can use them on vets-api.

Sending Metrics

Datadog automatically collects data sent to StatsD for each environment.

Metrics are sent out from the application via an API call to the statsd-instrument gem. There's a process called the StatsD Exporter which runs on each EC2 node that listens for events sent to it by the this library. That process is then queried by Datadog to gather metrics. There is generally no need for extra plumbing outside of simply starting to send metrics from within the application, with few exceptions.

An important thing to note while implementing metrics with this tool is that it is best-effort and should not be considered the source of truth for anything that must be 100% accurate (think billing report). It is extremely useful in measuring and detecting trends, but not a transactional database (or a database at all).

Metrics are stored for 15 months by default by Datadog.

Naming

Metrics names should not conflict with one another. Traditionally, they use dots—for example,  mhv.account.existed.

There's some transformation of names to other names plus tags in the StatsD exporter configuration for vets-api, but an understanding of this is not entirely necessary (nor is it necessary to do these translations to use StatsD to store metrics). This file also contains an alternative way to apply tags to metrics.

Tagging

Tags can be passed along with metrics as well, but a metric must consistently have the same set of tag names each time it is sent. If there is a mismatch, StatsD will start ignoring metrics sent that differ from the first style sent.

As an example of this, if we send a metric api.foo with tags value of {my_tag: bar}, a tag of name "my_tag" will be expected for any future metrics sent called api.foo. If we were to then send a metric named api.foo with tags value of {other_tag: baz}, StatsD will refuse the metric.

Care should also be made to not allow user input to enter tag values. Each combination of metric name + tag name/value pairs becomes a unique metric in Datadog.

Metric Types

There's documentation in statsd-instrument's page (linked above) so please review there for specific documentation for how to use them, but there are some special considerations when using these types with Prometheus the way that we do that we'll review here.

measure

StatsD.measure('my.measurement', 0.1234, tags: {'my_tag': 'value'})

Use this data type when measuring an amount of something. Examples would be the time something took or the size of a file being processed.

When using measure, Datadog will automatically translate the data into a few metric names and individual measurements will not be stored. First, it will store quantiles in a metric using the original name and a tag per quantile. By default it will give you 50%, 90%, and 99%. This metric is essentially "x% of things measured were under this value". Ask for help if you need different buckets. Secondly, it stores two additional metrics which are essentially like increments (below) suffixed with _sum and _count. These can be used for combining and averaging (see below for how to query this).

increment

StatsD.increment('my.counter', tags: {'my_tag': 'value'})

Use increment when you want to count occurrences of some event.

gauge

StatsD.gauge('my.gauge', 5, tags: {'my_tag': 'value'})

A gauge is used to record a value of something at a specific time (think speedometer in a car). Typically we'll want to use gauges only in a worker process as they are not likely to be event-based like the other types.

Surfacing Metrics in Datadog

Each node running the vets-api application runs its own instance of the StatsD exporter, which Datadog queries.

StatsD metrics that have been properly configured and have started seeing traffic can be found on the Datadog Metrics Summary page (SOCKS proxy required).

Once surfaced, that metric is available to be added to a Datadog dashboard.


JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.