Deployment Strategy

We intend to keep our repositories always deployable from the primary branch. The deployment schedule below describes when deployments typically happen, but deployments from the primary branch may occur at any time. https://www.va.gov does not use release branches; instead, we maintain a focus on keeping the primary branch healthy.

For finer control over when your checked-in feature is released, see feature toggles.

Automated Deploys

There are automated deploys of gibctvets-apiva.gov-cmsand vets-website Monday through Friday. The release is created from the primary branch at 2 pm Eastern and deployed at 3 pm Eastern. Code merged to the primary branch after 2 pm Eastern will not be added to a release and deployed until the next business day.

Exceptions to automated deploys

If there are days or periods of time when many people will be out of the office (i.e. holidays) the automated deploys will be suspended. If people aren’t in the office to support the code going out, it’s not responsible to release.

Note: This only applies to automated deploys. Engineer-initiated deploys (e.g. to address a critical bug) will be allowed to proceed as usual.

Holiday release freeze schedule

During holiday release freezes:

  • Automated deploys will not occur

  • All code deploys require approval via the "Requesting out-of-band-deploys" process below

  • Content-only deploys may continue without explicit approval, but need to be manually triggered

  • Limited support will be available from the Platform team.

There may be a change in this policy as VA dictates.

The holiday release freeze is in effect during the following dates:

2022

  • 05/30 - Memorial Day

  • 6/20 - Juneteenth (observed)

  • 07/04 - Independence Day (observed)

  • 09/05 - Labor Day

  • 11/11 - Veteran's Day

  • 11/24 - 11/27 - Thanksgiving (note: weekend included for clarity)

  • 12/24 - 1/2/2023 - Winter holiday freeze (note: weekends included for clarity)

Requesting out-of-band deploys

If there is a critical issue that needs to be resolved outside the automated deployment schedule, explicit permission must be granted for an out-of-band deploy.

Open an OOB Deploy Request Issue Ticket and verify each of the tasks.

For recommended approaches to issue resolution (i.e. when to revert, when to fix forward), see Resolving critical issues

For every out-of-band deploy requested, the Platform team will expect a follow-up postmortem from the requesting team, explaining the context that led to the problem and proposing follow-up actions to prevent similar future problems.

Is my issue critical?

Examples of critical issues include:

  • A bug that’s preventing a significant number of Veterans from accessing a feature

  • A bug creating a non-trivial deviation from expected functionality

  • A 508/accessibility failure of severity level 0 (Showstopper) or 1 (Critical) (severity rubric)

Examples of non-critical issues include:

  • Incorrect text or visual formatting that does not impede the feature from working

  • Any code for features not yet released to Veterans

  • Just wanting to get code out sooner

When in doubt on whether an issue is critical enough for out-of-band deployment, OCTO-DE leadership for the Platform will decide.

Step-by-step process for out-of-band deployment approval

  1. Open a support request in #vfs-platform-support using /support

    1. Assign to the team who manages the code needing deployment (Release Tools, Console Service Team, etc).

    2. Select “Off-Cycle Deployment“ for the nature of the request

    3. Include a summary of the issue, severity, and user impact

      1. User impact should include the number of users affected.

  2. Platform support staff triggers PagerDuty using /pd trigger. This will notify the relevant OCTO-DE Platform staff which is required for the next step

    1. Select “Out-of-band Deployment (service).

    2. Leave “Assign to” blank

    3. Put the summary, severity, and user impact from the Slack support ticket in the description, as well as a link to the #vfs-platform-support Slack thread.

  3. The deployment request must be approved (or declined) by OCTO-DE Platform staff before it may proceed.

  4. If approved, the Platform support staff will perform the deploy.

  5. Within two business days of this incident open a PR with a postmortem using this template and with the steps outlined in the included README.md.