StatsD Metrics
Last Updated: November 18, 2024
This document is an introduction to how StatsD metrics are sent and how a developer can use them on vets-api.
Sending metrics
Datadog automatically collects data sent to StatsD for each environment.
Metrics are sent out from the application via an API call to the statsd-instrument gem. The StatsD exporter process runs on each EC2 node and listens for events sent to it by this library. The process is then queried by Datadog to gather metrics. With a few exceptions, there is generally no need for extra plumbing outside of simply starting to send metrics from within the application.
Important: The metrics generated by the StatsD exporter are not 100% accurate.
While useful for detecting and measuring trends, StatsD exporter metrics should never be considered the source of truth.
By default, Datadog stores metrics for 15 months.
Naming
Metric names should not conflict with one another. Traditionally, metric names are formatted using dots; for example: mhv.account.existed.
There's some transformation of names to other names plus tags in the StatsD exporter configuration for vets-api, but an understanding of this is not entirely necessary (nor is it necessary to do these translations to use StatsD to store metrics). This file also contains an alternative way to apply tags to metrics.
Tagging
Tags can be passed along with metrics as well, but a metric must have the same set of tag names each time it is sent. If there is a mismatch, StatsD will start ignoring sent metrics whose styles differ from the first style sent.
For example, if we send a metric api.foo with tags value of {my_tag: bar}, a tag of name "my_tag" will be expected for any future metrics sent called api.foo. If we were to then send a metric named api.foo with tags value of {other_tag: baz}, StatsD would refuse the metric.
Care should be made to not allow user input to enter tag values. Each combination of metric name + tag name/value pairs becomes a unique metric in Datadog.
Metric types
The statsd-instrument's page provides documentation for using metric types, but there are some special considerations when using these types with Prometheus.
Measure
StatsD.measure('my.measurement', 0.1234, tags: {'my_tag': 'value'})
Use this data type when measuring an amount of something. Examples include the time a process took or the size of a file being processed.
When using measure, Datadog automatically translates the data into a few metric names; individual measurements are not stored. First, it stores quantiles in a metric using the original name and a tag per quantile. By default, it will give you 50%, 90%, and 99%. This metric is essentially "x% of things measured were under this value." Ask for help if you need different buckets. Secondly, it stores two additional metrics, which are essentially like increments (below) suffixed with _sum and _count. These can be used for combining and averaging (see below for how to query this).
Increment
StatsD.increment('my.counter', tags: {'my_tag': 'value'})
Use increment to count occurrences of some event.
Gauge
StatsD.gauge('my.gauge', 5, tags: {'my_tag': 'value'})
The gauge metric is used to record a value of something at a specific time (think speedometer in a car). Typically we'll want to use gauges only in a worker process, as they are not likely to be event-based like the other types.
Surfacing metrics in Datadog
Each node running the vets-api application runs its own instance of the StatsD exporter, which Datadog queries.
StatsD metrics that have been properly configured and have started seeing traffic can be found on the Datadog Metrics Summary page (SOCKS proxy required).
Once surfaced, that metric is available to be added to a Datadog dashboard.
Help and feedback
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.