Environments

Last Updated: November 21, 2024

This page describes both the production and non-production environments used during the development and deployment cycle for VA.gov.

DO NOT connect systems in the non-production environments to systems containing real PII/PHI.

Environment	Description	vets-api URL	vets-website URL
Development (dev)	A place to build and test code changes without breaking anything in the live production environment. Used by developers. Not connected to live end-user data.	Dev API	Dev homepage
Staging	Where code is deployed before it goes to production; often used for User Acceptance Testing (UAT) or to verify before releasing to production. Used by developers, QA engineers, and other VFS team members; sometimes used by end-users for UAT.	Staging API	Staging homepage
Sandbox	A copy of the staging environment typically used to define Service Level Agreements (SLAs) for clients developing against the VA.gov API (Lighthouse). Used by developers. Not connected to live end-user data. This is considered “Production” for the Lighthouse developer portal.	Sandbox API	none
Production	Live code used on VA.gov. Used by end-users visiting VA.gov (and other Veteran systems). Full production data. Not for experimentation or testing.	Production API	Production homepage

Where non-production environments live

The environments run in AWS Elastic Kubernetes Service (EKS), where Vets API bulkheads are deployed as Kubernetes deployments, which orchestrate and manage pods. See the Vets API on EKS documentation for further information.

Builds and deploys

For the dev, staging, sandbox, and production environments, merging to master in the vets-api GitHub repository automatically kicks off a set of GitHub actions to perform code checks, build and push a new container image to AWS Elastic Container Registry (ECR), and update the vsp-infra-application-manifests repository with the new image tag. Our continuous delivery tool, ArgoCD, detects the updated image tag and updates the Kubernetes deployment accordingly. Since Kubernetes is declarative, it reconciles the current state with the desired state. If a change is detected, ArgoCD initiates a sync process, gracefully rotating and pruning the deployments while launching new pods with the updated image and configurations.

For details on the daily production deployment, see Deployment process. Additionally, further details on Vets API on EKS can be found here.

Background job processing

The periodic_jobs.rb file (found here) in the vets-api repository defines Sidekiq jobs that run on a cron time-based schedule. To monitor job execution, use the Sidekiq UI to view queued or completed jobs. Worker instances in each environment handle the processing of these background jobs.

Unreleased feature testing

Our deployment process assumes that main/master should always be deployable. If you need finer control over the release of your feature, you can use feature toggles. Powered by the flipper gem, feature toggles are used to manage and preview unreleased features. Feature toggles allow for toggle switching without having to redeploy. In any environment, teams can enable or disable a feature for:

All users
A percentage of all users
A percentage of all logged-in users
A list of users
Users defined in a method

For information on using feature toggles, see Feature toggles.

Test users

Test users with various credentials and levels of assurance are available for the staging and development environments:

This user test account directory contains a listing of test user accounts for several VA.gov products.
This test accounts document provides a list of test users per identity provider (SOCKS proxy access is required to view this link).

ID.me provides authentication for users accessing services through VA.gov. Users don’t have a VA.gov-specific account. Instead, users sign in to ID.me and their credentials are passed to VA.gov. VA.gov uses this information to make additional requests for authorization within VA systems.

For more information about ID.me, see the FAQs on the ID.me support website. (You’ll need to create your own ID.me account to access their support information.)

Monitoring and error tracking

We use Sentry and Datadog to provide insights into performance metrics and error tracking. (SOCKS proxy access is required to access Sentry.) For Datadog access, see Monitor Applications and Infrastructure with Datadog.

A unique project exists in Sentry for each environment, including development.
Datadog includes some monitoring and alerting capabilities.

Database

Each environment is configured with a Postgres instance to store relational data. The database is managed by AWS RDS.

External service calls and mocked data

Environment	Data source
Development	Access to external services is blocked in development. Instead, responses come from the Vets API mockdata repo through the use of the betamocks gem, which returns pre-recorded responses.
Staging	In the staging environment, some responses are mocked and some live requests go through the forward proxy and connect to lower env backends. Mockdata comes from the Vets API mockdata repo through the use of the betamocks gem.
Sandbox	Access to external services is blocked in development. Instead, responses come from the Vets API mockdata repo through the use of the betamocks gem.
Production	Most of the data that is displayed, or stored from VA.gov lives in external services. In many cases, Vets API acts as a proxy for external services that live on the VA Network. These external services store information about Veteran’s health, benefits, education, and other records. Vets API connects to these external services via the forward proxy. In the production environment, live requests are made to external services. When appropriate, `vets-api` caches many of the responses from external services. When there is no cache record, a request is made through the forward proxy.

Breakers

Vets API uses the breakers gem to implement the circuit breakers pattern using Faraday middleware. Breakers is used in conjunction with external services to determine if a service may be down.

If an external service outage occurs, breakers record successful and failed requests in Redis to determine whether to mark a service as up or down. Instrumentation for breakers and external services can be found in Datadog.

Breakers are not used in the development or sandbox environment since no HTTP calls are made in those environments.

Logs

Logs for each environment are sent to Datadog. Vets API server and worker instances have their own logs for server or background job-related information.

PII and sensitive data

We expect developers to use the minimum amount of PII required by their application. Be aware of where and how that data is stored throughout the web request lifecycle. Most PII related to logs and Sentry errors gets filtered out automatically. vets-api stores very minimal PII, but if there is an absolute need to log PII, a PersonalInformationLog is available.

Do not store any PII in the development or sandbox environments.

For more information on PII, see Personal Identifiable Information (PII) guidelines.

How migrations are run

In each environment, Rails database migrations for vets-api are triggered by an automated process during the Vets API ArgoCD sync. Database migrations are executed through a Kubernetes job triggered by a pre-sync hook defined in the parent helm chart.

For more information about database migrations, see Database migrations.

Settings

You need to define appropriate config settings that vary in value for each environment in the relevant application-manifests(private repo) configurations. See this page for instructions.

Defaults for secrets that developers can use locally are also defined in the vets-api config/settings.yml and must be safe to provide to the public. Get help in Slack from the on-call DevOps engineer to arrange secure delivery and configuration for settings that are used in the development environment via AWS Parameter Store. This allows your team to manage its own secrets.

Help and feedback

Suggest content changes to this page.
Submit new Platform Website content.
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.