Vets-API Helm Charts, Environment Values, and SSM Parameters

Last Updated: September 1, 2026

Source of truth for how Vets-API is deployed on EKS. Live charts and values live in vsp-infra-application-manifests: parent templates in apps/vets-api-parent/, per-environment values in apps/vets-api/{dev,staging,sandbox,prod}/.

How to use

Use this page by the kind of work you are doing:

• Changing deployment configuration (replicas, probes, bulkheads, resources): start with Configuration, then the environment values table for the env you are editing.

• Adding or updating a secret / Setting: skip to How-To. Do not copy SSM values into Helm charts.

• An incident or a stuck sync: start with Troubleshooting.

• Understanding a specific Kubernetes object: use the parent manifest catalog.

Values were read from vsp-infra-application-manifests main on 2026-09-01. Image tags, replica counts, and ClamAV tags change constantly. Treat those rows as a snapshot. Confirm against the files linked in each section before you change anything.

Repo and environment map

Please consult the VA Platform documentation: https://depo-platform-documentation.scrollhelp.site/developer-docs/environments for a full map of of vets-api’s environments. Also see: https://depo-platform-documentation.scrollhelp.site/developer-docs/vets-api-on-eks#Vets-API-EKS-deploy-process to better understand the vets-api EKS deployment process.

Each env wrapper's Chart.yaml depends on the parent via repository: "file://../../vets-api-parent". ArgoCD renders the parent templates with that env's values.yaml. There is no separate GitHub Pages helm repo for Vets-API anymore.

Configuration

How a deploy actually ships

vets-api PR merges to master
  -> Once CI passes, GitHub Actions builds a Docker image and pushes it to ECR as vets-api:<git-sha>
  -> The same workflow commits that sha into apps/vets-api/<env>/values.yaml (image.tag)
  -> ArgoCD notices the manifests repo change and syncs
  -> Kubernetes pulls vets-api:<git-sha>

ArgoCD does not build Rails code. The image tag is the Rails release. A manifests-only change with the same tag is a configuration deploy: pods may roll, the same image keeps running.

Two useful rules:

• Sync without an image.tag change = configuration only.

• Sync with an image.tag change = new Rails code.

Source: https://depo-platform-documentation.scrollhelp.site/developer-docs/vets-api-on-eks#Vets-API-EKS-deploy-process

Chart layout

apps/
  vets-api-parent/          # shared templates + default values (mostly empty; env files fill them in)
    Chart.yaml
    values.yaml
    templates/*.yaml
  vets-api/
    dev|staging|sandbox|prod/
      Chart.yaml            # depends on file://../../vets-api-parent
      values.yaml           # nested under the key vets-api-parent:
      templates/secrets.yaml

Helm merges parent values.yaml with the env file. Empty parent keys are intentional defaults. Put real numbers in the env file, not in the parent, unless every environment should share them (ports, ClamAV sidecar shape, Datadog annotation template).

Do not copy parent templates into an env directory. Env wrappers should only add env-specific secrets.

ArgoCD sync order

ArgoCD applies resources in wave order. Negative waves run first. PreSync hooks run before the main sync. Resources with no hook run during the main sync (wave 0 unless annotated).

Env secrets.yaml objects are also PreSync at wave -3 (dev's extra SecretStore is -4). They must exist before db-migrate and before pods start, because both envFrom the ssm-env-vars secret.

db-migrate has no sync-wave annotation, so it is wave 0 of PreSync: after secrets and the ServiceAccount, before web/worker pods. backoffLimit: 1. A failed migration blocks the rest of the sync. That is deliberate: it is the safety net that keeps a bad migration from taking web pods.

Why PreSync exists. Without hooks, ArgoCD would apply ExternalSecrets, the ServiceAccount, db-migrate, and Deployments in one wave. Migrate would start before SSM secrets existed, fail, and someone would have to sync secrets by hand and rerun. PreSync plus waves force SecretStore -> secrets -> ConfigMaps -> SA -> migrate -> pods.

hook-delete-policy: BeforeHookCreation (and HookSucceeded on db-migrate) means ArgoCD deletes the previous hook object before creating the new one. ssm-env-vars uses refreshInterval: "0", so the secret is created once per hook life. A Parameter Store change does not show up in pods until the next sync recreates that ExternalSecret and the deployments roll.

Parent manifest catalog

These are the files in apps/vets-api-parent/templates/. Details follow the table.

ssm-secretstore.yaml (SecretStore)

• Purpose: Tells External Secrets Operator to read AWS SSM Parameter Store in us-gov-west-1, authenticated with the vets-api ServiceAccount (IRSA).

• Limitations: The ServiceAccount must already exist for later waves, but this wave runs before sa.yaml. In practice the SA already exists from the previous sync. A brand-new namespace would need a second sync, or the SA created some other way, before this store can authenticate.

environment-secret.yaml (ExternalSecret named ssm-env-vars)

• Purpose: Pulls every parameter under /dsva-vagov/vets-api/<target_env>/env_vars/ into a Kubernetes Secret named ssm-env-vars. Path rewrite:

  • Strip /dsva-vagov/vets-api/<env>/env_vars/

  • Convert remaining / to __

  • Replace other non-alphanumeric characters with _

So /dsva-vagov/vets-api/dev/env_vars/module/service/api_key becomes env var module__service__api_key. Web, worker, cron, and db-migrate containers all envFrom this secret.

• Limitations:

• • refreshInterval: "0": ESO will not poll SSM. Updates wait for the next PreSync recreation.

  • The find is a regexp over the whole env_vars prefix. A misspelled path (env_var singular, wrong env segment) simply never appears.

  • Keys must match Settings / config/settings.yml (Settings.module.service.api_key -> module__service__api_key). See Settings and Parameter Store.

  • This secret is string env vars, not files. Certs and .pem files go through app-secret-dir in the env secrets.yaml.

sa.yaml (ServiceAccount)

• Purpose. Creates the vets-api ServiceAccount with eks.amazonaws.com/role-arn so pods can assume the env IAM role (SSM read, and whatever else that role allows).

• Limitations. Only rendered when serviceAccount.enabled is true (it is, in all four envs). Role ARN is per-env.

settings-and-readiness-configmap.yaml

• Purpose. Renders ConfigMaps from common.configMaps. Used for files that must land in the pod as config, including readiness-related scripts if they are supplied that way.

• Limitations. Only emits objects when common.configMaps is populated. Empty in the current env files; most settings still come from SSM via ssm-env-vars.

lighthouse-pre-configs.yaml

• Purpose. ConfigMap for Lighthouse SAML / pre-config XML consumed by the app.

• Limitations. Data is still in Helm values (common.lighthousePreConfigs). There is an older normalization item to move this into SSM. Do not add new secret XML here.

ssoe-config.yaml

• Purpose. SSOe SAML ConfigMaps, only when web.ssoeEnabled is true.

• Limitations. Gated on that flag. Cert/key files still come from Parameter Store via secrets.yaml, not this template.

db-migrate-job.yaml

• Purpose. PreSync Job: bundle exec rails db:migrate using the new image tag, with ssm-env-vars already loaded. Runs before web/worker roll out.

• Limitations

• • restartPolicy: Never, backoffLimit: 1. One failure fails the sync.

  • Uses the same image as the coming deploy. A settings.yml typo or a missing SSM key often shows up here first.

  • Mockdata volume is only mounted when mockData.enabled is true.

  • No explicit sync-wave; relies on default 0 after secrets.

server-deployment.yaml

• Purpose: One Deployment per entry in web.webServices. That is the bulkhead pattern: vets-api-web plus optional dedicated deployments (vets-api-mobile, vets-api-vaos, and so on). Each pod typically runs three containers:

  • vets-api-web (Puma on 3000, stats on 9293)

  • ClamAV sidecar (when clamav.enabled)

  • socat-proxy (when web.socatProxyEnabled)

Datadog is a node/host socket mount (/var/run/datadog), not a third sidecar in the current templates. Rolling update uses web.maxSurge / web.maxUnavailable. Readiness is /app/bin/is_ready. Startup is HTTP /v0/startup_healthcheck on 3000. Liveness is disabled.

• Limitations:

• • Liveness is off on purpose. A liveness restart of a busy-but-alive Puma worker would drop in-flight work and can cascade. Do not enable it without an SRE review.

  • maxUnavailable: 0 in every env. Prod uses maxSurge: 1 (one extra pod at a time). Lower envs use maxSurge: 25%.

  • Preferred pod anti-affinity only (affinityWeight: 100). Kubernetes may still colocate pods.

  • revisionHistoryLimit: 1.

  • Mockdata initContainer clones vets-api-mockdata with GIT_SSL_NO_VERIFY=true (cluster TLS workaround). Prod has mockData.enabled: false.

  • Parent comment on ClamAV: enabling it incorrectly in a copied chart can take production down. For Vets-API it is enabled as this sidecar, using the vsp-infra-clamav image.

worker-deployment.yaml

• Purpose. Sidekiq deployment vets-api-sidekiq, gated on worker.enabled. Same image, same ssm-env-vars, own HPA metric (sidekiq-utilization).

• Limitations. Only one worker deployment. No bulkheads for queues. Autoscaling is thread-busy based, not CPU. The Sidekiq DatadogMetric query inlines RAILS_MAX_THREADS from worker.envSecrets when Helm renders; if that env is missing, the HPA metric is wrong.

cron-deployment.yaml

• Purpose. CronJobs from cronJobs. Currently expiry-scanner (0 12 * * MON) in all four envs. concurrencyPolicy: Forbid.

• Limitations. Same image and secrets as web. A cron that needs a secret not in ssm-env-vars will fail the same way a web pod would.

server-service.yaml

• Purpose. ClusterIP Service per webService. Ports 3004 -> web (3000) and 9293 -> stats.

• Limitations. Parent values say do not enable sticky sessions. Sticky sessions break rolling deploys and HPA drain.

ingress_route.yaml

• Purpose. Traefik IngressRoute plus real-IP middleware. Loops web.webServices.*.routes. Match is Host(...) && <path>.

Traffic context:

• • *.va.gov goes through revproxy before the cluster.

  • *.vfs.va.gov goes straight to EKS.

• Limitations. Priority resolves overlapping routes. Catch-all PathPrefix(\/`)onvets-api-webis priority 10; bulkhead prefixes are 30/40. A new bulkhead route with a default/low priority will lose to the catch-all. Duplicateautoscaling:keys in a webService block (prodvets-api-mobile` has this) mean YAML keeps the last one.

external-dns.yaml

• Purpose. DNSEndpoint CNAME for web.externalDns (for example prod platform-api.vfs.va.gov -> traefik-prod.vfs.va.gov).

• Limitations. Only as useful as the ExternalDNS controller in the cluster. TTL is 180s in current values.

hpa.yml and hpa-external-metric.yaml

• Purpose. One HPA per webService with autoscaling.enabled, plus a DatadogMetric puma-backlog-<service>. Worker gets sidekiq-utilization when worker autoscaling is on.

Web scaling is Puma thread saturation, not CPU or memory:

puma-backlog = percent of (max_threads * ready replicas) that are busy

HPA targetValue is that percent. Worker metric is sidekiq.busy / (replicas * RAILS_MAX_THREADS) * 100.

• Limitations:

• • Requires DatadogMetric CRDs and a working Datadog cluster agent. If the metric is missing, HPA will not scale (it will not fall back to CPU).

  • Scale-down is conservative: stabilization windows, preStop sleep (~90s from parent sleepGracePeriodSeconds), successThreshold: 2 on readiness, and MIN_READY_PODS inside is_ready. Expect over-provisioning after a traffic spike.

  • Target values differ by env and bulkhead (prod web default target 30, some bulkheads 20; lower envs often 60).

  • Sidekiq DatadogMetric inlines RAILS_MAX_THREADS from worker.envSecrets at template time. Changing thread count in SSM alone will not update the HPA query.

postman-test.yaml

• Purpose. Helm test pod (helm.sh/hook: post-install,post-upgrade) running the postman image against domain.

• Limitations. This is a Helm test hook, not an ArgoCD PostSync hook. It does not gate ArgoCD syncs unless someone runs helm test.

Environment values

Values files: dev, staging, sandbox, prod.

Snapshot from 2026-09-01.

Shared shape

All four files nest overrides under vets-api-parent: and set:

Cross-env differences

From 9/2026

Prod bulkheads

Catch-all traffic stays on vets-api-web. Higher-priority Traefik paths send latency-prone or noisy routes to their own Deployment + HPA. See the webServices section of vets-api/prod/values.yaml for the most up-to-date information. Info below is a snapshot.

Metrics: Vets-API scaling and Puma backlog (requires Datadog access).

Staging bulkheads: vets-api-web (catch-all, default HPA 2/40 target 60), vets-api-mobile (2/32, target 40), vets-api-my-health (2/32, target 40). Dev and sandbox are single-bulkhead (vets-api-web only).

Parent values.yaml is the schema: empty keys that each env must fill (resources, replica counts, image.tag), plus shared defaults (web.sleepGracePeriodSeconds: 90, liveness off, ports 3000/9293, clamav.enabled: false until an env turns it on). All four envs currently set clamav.enabled: true. Do not copy numbers from this snapshot into a PR without re-reading the env file.

RDS constraint. Each bulkhead's Puma threads times pods consume Postgres connections. The RDS instance limit has been raised over time (now 5,000). Rolling deploys temporarily raise pod count (maxSurge). You cannot add a bulkhead per module without checking connection math.

SSM parameters and ssm-env-vars

String secrets and env-specific Settings are not Helm values. They live in AWS SSM Parameter Store and are loaded by the parent chart's environment-secret.yaml into a Kubernetes Secret named ssm-env-vars. Web, worker, cron, and db-migrate all envFrom that secret.

CLI create/update/delete of a parameter (any platform service, including Vets-API): Store a secret in Parameter Store.

Vets-API Settings conventions and settings.yml mapping. See this page for more information: Settings and Parameter Store. In ArgoCD, look for the Kubernetes Secret named ssm-env-vars.

Path convention:

/dsva-vagov/vets-api/<env>/env_vars/<setting/path/with/slashes>

Every Setting that uses an ENV in settings.yml needs a parameter in all four deployed environments, even if unused, so a missing key cannot boot-loop one env.

Secret files (certs, keys, pems) are a different SSM tree and a different Kubernetes Secret:

/dsva-vagov/vets-api/<env>/<file_name>

Mounted at /srv/vets-api/secret/<file_name> via app-secret-dir (and related ExternalSecrets) in the env secrets.yaml. Hardcode that path in settings.yml so you do not need a second parameter for the path.

See this documentation for more information:

• Settings and Parameter Store (settings.yml <-> SSM path)

• Store a secret in Parameter Store (CLI: add, update, retrieve, delete)

• How to add secret files to Vets-API (certs/keys mounted at /srv/vets-api/secret)

Dev secrets.yaml still has a new-cluster vs old-cluster split (currentKubeMinorVersion). New secret files in dev must be added to both ExternalSecret lists.

How-To / Runbooks

Render the charts locally (smart to do before submitting a manifests PR)

Parent charts already live on main as a file dependency. You do not need the old temp-vets-api-parent-helm-charts branch or a GitHub Pages helm repo.

brew install helm   # once
cd vsp-infra-application-manifests/apps/vets-api/dev
rm -rf charts Chart.lock
helm dependency build
helm template . --debug

Repeat for staging, sandbox, and prod if your change touches values or parent templates. helm template does not talk to the cluster. It will not catch a missing SSM parameter.

If helm dependency build pulls a .tgz instead of the local parent, check Chart.yaml still says repository: "file://../../vets-api-parent".

Change a Helm value (replicas, probes, routes, resources)

1. Edit apps/vets-api/<env>/values.yaml under vets-api-parent:.

2. If the change belongs in all envs and is truly a default, edit apps/vets-api-parent/values.yaml instead, then confirm each env does not override it.

3. Render locally.

4. Open a PR in vsp-infra-application-manifests. This does not ship new Rails code unless image.tag also changes.

Adding a bulkhead: add a web.webServices.<name> block with routes (priority > 10 so it wins over the catch-all) and autoscaling. Check RDS connections. If there aren’t enough, do not add a bulkhead.

Add or update a Setting (env var via SSM)

Follow Settings and Parameter Store:

1. Add the key to config/settings.yml as <%= ENV['module__service__key'] %> and to development.yml / test.yml with fake values.

2. Create the SecureString in all four envs at /dsva-vagov/vets-api/<env>/env_vars/module/service/key.

3. Wait for the next ArgoCD sync or ask #vfs-platform-support to restart the deploy if it is urgent.

SSM writes do not roll pods by themselves. ssm-env-vars is only rebuilt on PreSync.

Add a secret file (cert, key, pem)

Follow How to add secret files to Vets-API:

Recycle pods without a new image

Pods replace when the Deployment is OutOfSync (spec changed) or when you restart it. Re-syncing ArgoCD with no spec change will not recycle pods. Parameter Store updates will not recycle pods.

kubectl rollout restart Deployment/vets-api-web --selector=app=vets-api-prod

Use the env's actual deployment names and label selector. Please open a support issue in #vfs-platform-support for help with recycling the pods.

Rails console / Argo terminal

Please refer to this documentation for full instructions on using the terminal as well as steps for membership and access.

Troubleshooting

Sync succeeded but Rails code did not change

Check image.tag in the env values.yaml in vsp-infra-application-manifests. If it is the same SHA, you deployed configuration only. New pods still run the old image.

db-migrate failed / web never rolled

Check the logs (in Argo) of the failing pod. The migrate Job uses ssm-env-vars, so a new Setting that is not in Parameter Store will fail here before traffic moves. Reach out to Platform Support if you notice a failed migration.

Setting is empty or wrong in the pod

1. Is everything spelled correctly? env_vars (plural), double underscores in the ENV name, slashes in the SSM path matching Settings dots.

2. Confirm the parameter exists in the same target_env.

3. Confirm a sync happened after the SSM write. Look at the ssm-env-vars Kubernetes Secret in ArgoCD (keys only).

4. If the Secret has the key but the process does not, the pod has not rolled.

Secret file not found at /srv/vets-api/secret/...

Mismatch between SSM path, secrets.yaml secretKey, and settings.yml. In dev, the file may be on only one of the two ExternalSecrets.

499 / 502 while HPA shows spare capacity

Autoscaling is thread saturation, not CPU. is_ready can keep an overloaded pod in the load balancer when replica count is at MIN_READY_PODS (default 3) to avoid cascading failure. Spare CPU does not mean spare Puma threads.

HPA not scaling

Check the DatadogMetric puma-backlog-<deployment> (or sidekiq-utilization) in ArgoCD. If the query is broken (wrong target_env / cluster name), HPA has no signal.

Do not enable

• Liveness probes on web (disabled on purpose).

• Sticky sessions on the Service.

• ClamAV in a chart that is not this Vets-API sidecar pattern.

• Deleting an SSM parameter that is still referenced in settings.yml or secrets.yaml. Make sure the code that removes all references is deployed, then delete.

Reference

Probes and HPA

Every web container:

Startup probe must pass before readiness is evaluated. is_ready (every 5s in lowers):

1. Read local /stats. Failure -> not ready (Puma is down).

2. If busy-thread percent is below AUTOSCALING_TARGET_VALUE (often 60) -> ready.

3. If busy and Datadog says replicas <= MIN_READY_PODS -> stay ready (protect the cluster).

4. If busy and extra replicas exist -> not ready (leave the load balancer).

5. If Datadog is unreachable -> stay ready (fail open).

preStop sleeps ~90s so in-flight requests drain. Combined with HPA scale-down windows, the cluster sheds pods slowly on purpose.

Related Developer Docs

• Vets API on EKS

• Settings and Parameter Store

• How to add secret files to Vets-API

• Deployments

• Store a secret in Parameter Store

Help and feedback

• Suggest content changes to this page.

• Submit new Platform Website content.

• Get help from the Platform Support Team in Slack.

• Submit a feature idea to the Platform.