Review instances allow testing against live pre-production backends. If you need to do additional testing outside of local machine testing, or a test backend change, review instances are the best option. Review instances are a deployment of a combination of vets-api and vets-website and the ngnix forward proxy at specified versions.

Review instances are created as part of a pull request for the vets-api or vets-website GitHub repositories, but they can also be created manually by running a Jenkins job.

Where it lives

Review instances are launched in their own Virtual Private Cloud (VPC) in the vetsgov-staging VPC.

Builds and Deploys

The Jenkins files in vets-website and vets-api define a stage that triggers a review instance deployment. Opening a PR will trigger this process, which will generate a "GitHub Deployment" for the PR. A message on the PR will provide a link to the review instance.

Review instances are a deployment of a combination of vets-api, vets-website, & the nginx reverse proxy at specified versions.

Code is re-deployed on each commit which could result in lost changes local to the review instance.

Review instances are built by Jenkins and run on an independent virtual machine on which both vets-api and vets-website are installed. A nginx proxy in the frontend directs requests. The web interface (vets-website) and API (vets-api) are available over HTTP behind the SOCKS proxy.

The review instances are accessible via SSH and you can modify code and see the changes.

Cleanup

A Jenkins job will run periodically and remove review instances for which the source branches no longer exist. To ensure that your instance is cleaned up appropriately, just delete the branch from the origin repository.

The review instance is deleted when the non-main branch that a review instance is related to is deleted or when the instance is older than 7 days.

Feature Toggles

Feature toggles, powered by the flipper gem, can be used to manage unreleased features in a CI environment. Teams can enable or disable a feature for all users, a percentage of all users, a percentage of all logged-in users, a list of users, or users defined in a method. Feature toggles allow for toggle switching without having to redeploy. To enable to disable a feature, the development environment has a distinct url. Since review instances are launched in the development environment they will utilize the feature toggles defined here: https://dev-api.va.gov/flipper/features

Test Users

A variety of test users are available for the staging and development environments with various credentials and levels of assurance. This directory contains a listing of test user accounts for several VA.gov products. Test users defined for the staging environment can be used for review instances since they point to the staging environment.

Sentry and Grafana

Review instances do not report errors to Sentry and Grafana.

Database

Review instances run postgres from the postgres docker image locally on the review instance machine itself.

External Service Calls vs Mocked Data

vets-api connects to many external services via the forward proxy. This contrasts with the development and staging environments in which external service responses are either mocked or call out to a lower environment. Review instances are connected to pre production backends and live in the vetsgov-staging VPC.

Authentication

ID.me SAML configuration requires explicitly defining a callback URL via a manual process. This callback URL receives authentication information after a successful authentication process. Review instances require a special nginx configuration that intercepts the callback to the staging-api.va.gov server, and forwards the authentication information to the appropriate review instance. The information is mapped by the RelayState parameter, which is provided to the review instance vets-api config with the REVIEW_INSTANCE_SLUG environment variable).

For implementation-specific details, see the revproxy nginx configuration.

This authentication sequence diagram is a visual of the review instance authentication.

Logs

Review instance logs can be pulled from AWS and analyzed. Log events can be queried via Cloudwatch or Cloudwatch Insights. Logs exist within the container on individual review instance machines.

PII and sensitive data

Review instances should not store PII.

The term "personally identifiable information" refers to information which can be used to distinguish or trace an individual's identity, such as their name, social security number, biometric records, etc. alone, or when combined with other personal or identifying information which is linked or linkable to a specific individual, such as date and place of birth, mother’s maiden name, etc.

We expect developers to use the minimum amount of PII required by their application and be aware of where and how that data is stored throughout the web request lifecycle.

Most PII related to logs and Sentry errors gets filtered out automatically. vets-api stores very minimal PII, but if there is an absolute need to log PII, a PersonalInformationLog is available.

How migrations are run

Review instance deployments will automatically trigger database migrations.

For more information on database migrations, see vets-api database migrations

Settings

Review instances use their own separate configuration settings defined here and here.

Review instances also use their own separate settings defined here