The development and deployment cycle for VA.gov makes use of various environments.
Where non-production environments live
Environments each live in their own Virtual Private Cloud (VPC). Each of these environments also has their own Jumpbox--an EC2 instance that lives on the firewall border of the VPC and allows access to servers within the VPC. Our provisioning configuration deploys the API as the
vets-api, which runs an HTTP server, and
vets-api-worker, which processes background jobs.
dev environment deployment lives in AWS Elastic Kubernetes Service (EKS) where vets-api is deployed as pods vs a standalone docker container deployed to EC2 instances.
Builds and deploys
dev, merging to master in the
vets-api GitHub repository automatically kicks off a set of GitHub actions that will update the special branch (
k8s) we are currently using for Kubernetes-based deployment specifications, build and push a new container image to AWS Elastic Container Registry (ECR), and update the
application-manifests repository with the new image tag. From here, our continuous delivery tool, ArgoCD, will see the updated image tag and update the EKS deployment by standing up new pods using the new image and taking down the old ones.
staging, merging to master in the
vets-api GitHub repository automatically kicks off a build in Jenkins and the new changes are then deployed to to EC2 instances within the
production environment, code is deployed daily, Monday - Friday by a cron-based deployment pipeline managed within Jenkins.
Background job processing
There are a number of Sidekiq jobs defined in
sidekiq_scheduler.yml in the
vets-api repository that run on a time based cron job schedule. If an issue occurs with a job, or to view which Sidekiq jobs have been queued or completed, see this Grafana chart. There are worker instances for each environment which process the background jobs.
For help troubleshooting, see Sideqik jobs.
Unreleased feature testing
Our deployment process assumes that master should always be deployable. If you need finer control over the release of your feature, you can use feature toggles. Powered by the flipper gem, feature toggles are used to manage and preview unreleased features. Feature toggles allow for toggle switching without having to redeploy. In any environment, teams can enable or disable a feature for:
a percentage of all users
a percentage of all logged-in users
a list of users
users defined in a method
A variety of test users are available for the staging and development environments with various credentials and levels of assurance.
This test accounts document provides a list of test users per identity provider
ID.me provides authentication for users accessing services through VA.gov. Users don’t have a VA.gov-specific account. Instead, users sign in to ID.me and their credentials are passed to VA.gov. VA.gov uses this information to make additional requests for authorization within VA systems.
Monitoring and error tracking
A unique project exists in Sentry for each environment, including development.
Grafana includes a collection of resources including dashboards with a variety of metrics and visualization options. The “production” Data Source can be selected for the ability to view metrics in the development environment
Each environment is configured with a Postgres instance to store relational data. The database is managed by AWS RDS.
External service calls and mocked data
In the staging environment, some responses are mocked and some live requests go through the forward proxy and connect to lower env backends. Mockdata comes from the Vets API mockdata repo through the use of the betamocks gem.
Vets API connects to these external services via the forward proxy. In the production environment, live requests are made to external services. When appropriate
Vets API uses the breakers gem to implement the circuit breakers pattern using Faraday middleware. Breakers is used in conjunction with external services to determine if a service may be down.
If an external service outage occurs, breakers record successful and failed requests in Redis to determine whether to mark a service as up or down. Instrumentation for breakers and external services can be found in Grafana.
Breakers are not used in the development or sandbox environment since no HTTP calls are made in those environments.
Logs for each environment are sent to Loki. For a more granular level of analysis, per instance logs can be analyzed. Log events can be queried via Loki, leveraging the Grafana interface. Vets API server and worker instances have their own logs for server or background job related information.
PII and sensitive data
We expect developers to use the minimum amount of Personal Identifiable Information (PII) required by their application. Be aware of where and how that data is stored throughout the web request lifecycle. Most PII related to logs and Sentry errors gets filtered out automatically.
vets-api stores very minimal PII, but if there is an absolute need to log PII, a
PersonalInformationLog is available.
Do not store any PII in the development or sandbox environments.
How migrations are ran
In each environment, Rails database migrations for
vets-api are triggered by a manual process.
You will need to define appropriate config settings that vary in value for each environment in the relevant devops (private repo) and application-manifests(private repo) configurations (
prod-settings.local.yml.j2 in the
devops repository and yaml configurations with the
dev directory of the application in the
Defaults for secrets that developers can use locally are also defined in the vets-api
config/settings.yml and must be safe to provide to the public. Get help in Slack from the on-call Platform-Infrastructure engineer to arrange secure delivery and configuration for settings that are used in the development environment via AWS Parameter Store (this step has been heading in the direction of and will continue to move towards a more self-service model by allowing teams to manage their own secrets).
Help and feedback
Create an issue ticket to suggest changes to this page