Skip to main content
Skip table of contents

Deployment process to production

Last Updated:

Introduction

To ensure code stability and quality, code goes through several steps to get to production. The release process as well as the rationale for that process are outlined below.

The same deployment process applies for both content and code changes.

Automated Deployment Schedule

See the automated deployment schedule table.

Release History

Process overview

For feature and hotfix development for vets-website and content-build (code changes and content changes), see the steps below. For vets-api process, see the Vets API EKS deploy process detailed here.

  1. Dev/Content: Create feature branch from main branch

  2. Dev/Content: Commit changes to feature branch

  3. Dev/Content: Feature branch merged to the main branch (via Pull Request)

  4. Automatic: Build run from main branch to create an artifact

  5. Automatic: Deploy newly created artifact to dev and staging

  6. Automatic: Create a release in GitHub from main and tag artifacts of that commit with release name

  7. Automatic: Deploy to production using artifacts

Process details

1. Dev: Create feature branch from main branch

2. Dev: Commit changes to feature branch

3. Dev: Pull Request is opened and once reviewed and approved, feature branch is merged to the main branch.

  • GitHub Actions runs unit tests, linting, and security scans.

Committing and code reviewing developers are responsible for running automated and manual integration tests locally before closing the pull request

4. Automatic: Build run from main branch to create an artifact

5. Automatic: Deploy newly created artifact to development and staging environments

  • All VA.gov GitHub repos are set up to do squash merges (via the GitHub PR interface), leaving behind a clean revision history that is feature based.

The main branch should always be deployable. As such, the deployment to the staging environment is configured to happen automatically and can be used to see what something would look like in a production-like environment for any kind of manual testing/verification.

Because main is designed to always be deployable, long-running features that should not be deployed should utilize feature toggles in the code that disable the feature for the production environment. It's likely that breakages in staging will occur and that this is necessary to discover these prior to moving anything to the production steps.

6. Automatic: Create a release in GitHub from main and tag artifacts of that commit with release name

  • Every weekday, at the beginning of the daily deployment, a Jenkins or GitHub Action (for vets-website and content-build) automerge job sends a link to the #status-vets-website or #status-content-build Slack channels with the status. This commit reference is stored to ensure the diff and released version are deterministic.

7. Automatic: Deploy to production using artifacts

  • From here, Jenkins or Github Actions can kick off a production deployment. After the deployment occurs, the normal site monitoring infrastructure will be used to validate it is working. As this process is automatic any new features should have monitoring in place before, or as a part of their deployment

The code that appears in the main branch actually gets deployed to both dev and staging environments. This is done to support different configurations for the DevOps team as they work to support any configuration changes (i.e. in dev first).

Deployment Rollbacks

If a production deployment introduces issues that affect Service Level Objectives (SLOs) established for the project, a Platform team may restore service to users by rolling back the deployment. This is accomplished by triggering a new deploy job in Jenkins or Github Actions using a previous release tag or commit.

The use of hotfixes is discouraged, but may be useful in an emergency situation when main has significantly deviated from the release and a fix to the failed production release is critical. To create a hotfix, create a branch from the last stable release tag, make changes necessary (with review), create a new release tag following the correct naming scheme, and trigger a deploy in Jenkins with the release name as a parameter.

If SLOs are not affected and a fix is not critical, no rollback will be issued. Instead the fix should be applied through the standard development workflow.

Background and context for deployment process

The creation of this deployment process was triggered and influenced by the following problems (expected or currently experienced) and feedback (based on previous drafts).

Complexity of branching strategy

This concern hovers around two things:

  • Will people be able to do it without much learning or frequent git mistakes?

  • Is our approach an industry standard such that it has a low learning curve?

The process originally followed most closely aligned with GitHub Flow, but this process proved to be overly simple for multiple projects being committed to our vets-website and vets-api repositories.

Other popular flows such as Git Flow and GitLab Flow were suggested, but have the opposite problem. While being industry standards (like GitHub Flow), they tend to be more complex than our team's needs demand. They are complex enough that they require expert Git knowledge, merging mistakes are commonly made by teams that use them, and they require more active management to maintain clean merge trees. Git Flow and GitLab do some things really well though: clear guidance on what is being developed and what is in which environment.

While it's not ideal to create our own strategy, none of the strategies seemed to fit the bill with priority being on simplicity. Instead this one was written and does the following things to meet its goals:

  • Stays very close to GitHub Flow, the simplest of the industry standards and the easiest to learn/manage

  • Uses Git Flow's emphasis on main always being deployable, which eases deployment and rollback

  • Seven repeatable steps for any kind of development (feature or hotfix)

  • Integrates our team's people work flow (i.e. interactions between development and Platform teams)

Different flows between features and hotfixes

This is one of the weaknesses with Git Flow, which requires additional management. We've simplified this by creating a process that is the same for both features and hotfixes. It's expected that things will go to production at least weekly, meaning that there shouldn't be any feature or hotfix branches to manage for long periods of time.

Off-schedule Deployments

For out-of-band deployment information, see our /wiki/spaces/pilot/pages/1279787009.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close