Triaging API Errors in EKS Environments

Last Updated: October 21, 2026

When troubleshooting errors in vets-api there are several tools available.

• Datadog for logs, error tracking, and metrics

• ArgoCD for deployment errors

This page explains how to investigate errors in those places.

Datadog logs

Detailed logs for Vets API for all EKS environments are stored in Datadog.

• EKS Dev Logs

• EKS Staging Logs

• EKS Sandbox Logs

• EKS Prod Logs

Logs can be filtered by container, pod, urls, IP addresses, etc.

Datadog error tracking

Datadog provides an error tracking framework.

ArgoCD logs

ArgoCD container logs can be viewed in the Logs tab inside each pod. Logs can be viewed, copied, downloaded, and followed.

Long term error tracking

See also the StatsD Metrics page.

Tracking totals from the api:

StatsD.increment("service.method.total")

For failures we can add tags to differentiate error types within failure totals:

StatsD.increment("service.method.fail", tags: ["error:#{error.class}"])

The above pattern is common enough in service classes that it's been abstracted out to a concern, Common::Client::Monitoring, which can be mixed in to a service.

module EVSS
  class Service < Common::Client::Base
    include Common::Client::Monitoring

Service calls can then be wrapped in a block that automatically records totals and failures:

def get_appeals(user, additional_headers = {})
  with_monitoring do
    response = perform(:get, '/api/v2/appeals', {}, request_headers(user, additional_headers))
    Appeals::Responses::Appeals.new(response.body, response.status)
  end
end

With those calls in place we can query for the average across deployed server instances in Datadog:

avg:vets_api.statsd.api_appeals_get_appeals_total{env:eks-prod}.as_count()

The error query can filter by error tag:

sum:vets_api.statsd.api_appeals_get_appeals_fail{env:eks-prod} by {error}.as_count()

Help and feedback

• Suggest content changes to this page.

• Submit new Platform Website content.

• Get help from the Platform Support Team in Slack.

• Submit a feature idea to the Platform.