Skip to main content
Skip table of contents

Downtime notifications

Last Updated:

Downtime Notifications are a mechanism for handling outages of external services in the consuming front-end applications.

Overview

  • Available as a React component in vets-website.

  • Consumers of the React component register required-services as dependencies via React props

  • Driven by PagerDuty maintenance windows so that downtime for a certain service can be scheduled without requiring a deployment. More info on PagerDuty maintenance windows.

  • After a maintenance window is added in Pager Duty, an application wrapped in the Downtime Notification React component typically responds in one of three ways:

    1. Rendering the application as normal if the maintenance window does not start for upwards of an hour.

    2. Displaying a dismissible modal informing the user of upcoming downtime and rendering the application as usual if the maintenance window starts within the hour.

    3. Rendering an alert banner informing the user of active downtime instead of rendering the application if the maintenance window has already started.

How to use

Prerequisites

  • Identify required-services for the application

    • These services should be available in externalServices if they have not already been. The value must match that in the vets-api manifest.

    • These dependencies may not be directly accessed by your application, but are more likely to be dependencies of the API method(s) consumed by your application. For example, the Search app requires Search.gov to function, even though it doesn't directly hit Search.gov.

Forms library applications

  1. Import the services enum

    CODE 
    import { externalServices } from 'platform/monitoring/DowntimeNotification'

  2. Add a downtime property and dependencies array to the root of the form config. For example:

    CODE 
     downtime: {
    dependencies: [externalServices.evss, externalServices.emis, externalServices.mvi, externalServices.vet360],
  },

  3. Add additional properties to downtime as needed:

    1. message

      1. a react component displayed in place of the default downtime message

      2. custom downtime messages are discouraged

    2. requiredForPrefill

      1. true:

        1. given a user is logged in, when any services in the dependencies array are down, then display downtime notification before starting a form

        2. given a user is logged in and has a saved form, when any services in the dependency array are down, then display the downtime notification before application submission

        3. given a user is not logged in,

      2. false [default]: when any services in the dependencies array are down, then display the downtime notification before application submission

  4. If you need your FormBuilder app to have downtime notifications on intro page you also need to follow React implementation below.

    1. The above implementation only adds downtime notifications to the review page. Most forms will probably require to also the notifications on intro page so users don't find out at the end of the form.

The DowntimeNotifications component wraps the FormStartControls on the introduction page and the SubmitController on the review page. Users can edit and save forms for later during a downtime unless requiredForPrefill is true.

React applications

  1. Import the Downtime Notification React component into the application React code, usually the topmost container.

  2. In the render method, render a Downtime Notification React component passing the identified dependencies as props and the affected components as React children.

Example

Suppose an application My App leverages a service called Fancy Service, and Fancy Service was known to be offline for maintenance now and then. During those timeframes, it is desirable to show a message to the user explaining that My App is unavailable at this time so that users aren't given the impression that they are running into unexpected errors and so that Fancy Service isn't needlessly hit. This should also be easy for engineers to do, without requiring any deployments to publish/unpublish the messaging. This is a pretty typical use case for the DowntimeNotification React component.

React code

First, the topmost React container of My App is modified to import and consume the Downtime Notification React component.

Note that MyAppDataGrid would issue the API request to Fancy Service during componentDidMount, rather than MyApp. This is because MyAppDataGrid is passed a child component to DowntimeNotification. DowntimeNotification will fetch the currently-registered maintenance windows from the VA API (which in return reads the maintenance windows from PagerDuty), and if it discovers active downtime for fancyService, then the componentDidMount lifecycle method of MyAppDataGrid will not execute. This way, API requests to Fancy Service won't continue to send during downtime.

src/applications/my-app/containers/MyApp.jsx

CODE 
import React from 'react';
import { connect } from 'react-redux';

import {
  DowntimeNotification,
  externalServices,
} from '@department-of-veterans-affairs/platform-monitoring/DowntimeNotification';

import { fetchFromFancyService } from '../actions';

import MyAppDataGrid from '../components/MyAppDataGrid';

class MyApp extends React.component {
  render() {
    return (
      <>
        <h1>My App</h1>
        <DowntimeNotification
          appTitle="my app"
          dependencies={[externalServices.fancyService]}
        >
          <MyAppDataGrid
            fetchFromFancyService={this.props.fetchFromFancyService}
          />
        </DowntimeNotification>
      </>
    );
  }
}

const mapStateToProps = state => store.myApp;

const mapDispatchToProps = {
  fetchFromFancyService,
};

export default connect(mapStateToProps, mapDispatchToProps)(MyApp);

The render-flow for this container is:

  1. MyApp's render method is executed

  2. DowntimeNotification issues a GET request to the VA API for all registered maintenance windows.

  3. DowntimeNotification processes the response JSON data into a map.

  4. DowntimeNotification searches the downtime map for a service called fancyService, as passed in its dependencies prop.

  5. If downtime for fancyService is found:

    1. If the current time is in the timeframe of the downtime window, render an alert banner informing the user that this application is undergoing maintenance.

    2. Otherwise, DowntimeNotification will render its children, in this case MyAppDataGrid. A dismissible modal is also rendered if downtime is approaching within the hour.

What the notification looks like one hour before service goes down with no custom content

Downtime notification modal

Downtime notification an hour prior to service going down

This is what the downtime notification looks like during downtime period

Downtime notification alert

Downtime notification during the downtime period

Adding a new PagerDuty service

If you've created a new backend service that a React application depends on, follow these steps to allow the React application to fetch its downtime windows:

Configuration

  1. Login to PagerDuty (https://dsva.pagerduty.com/)

  2. Create your PagerDuty Service (https://dsva.pagerduty.com/service-directory)

    1. This may require you to create a PagerDuty Person, Escalation Policy, and/or Team that is specific to this service.

    2. Follow the naming convention of other backend services in PagerDuty by prepending "External: " to the service name (i.e. "External: MyService").

    3. You may also want to create multiple services for multiple environments (i.e. "External: MyService", "Staging: External: MyService", "Dev: External: MyService").

  3. In the Application Manifest repo, add your service to the list of maintenance services in the values.yml file for each vets-api environment.

  4. Deploy all changes.

Testing

  1. Login to PagerDuty (https://dsva.pagerduty.com/)

  2. Go to your staging service

  3. Create a maintenance window sometime in the future.

    1. You can specify a message to a user by using the token "USER_MESSAGE: " in the description (i.e. "Down for v16 update. USER_MESSAGE: We'll be back soon!")

      1. Note that custom messages are discouraged to maintain a consistent user experience.

  4. Visit https://staging-api.va.gov/v0/maintenance_windows and ensure that your service's downtime window is listed

    1. This information is cached and may be delayed by up to 3 minutes (see the PagerDuty::PollMaintenanceWindows job).

Other examples

  • The Search app is a basic example of an application having a single service dependency, in this case, http://Search.gov. The Facility Locator is another example.

  • The Sign-In Modal is an example of a component that renders messaging about service downtime. This does not affect the component's functionality; instead, it's more of a heads-up for the user about potential difficulty. This is also the case for Letters.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close