Flaky test management in Cypress

Flaky tests are tests that sometimes pass and sometimes fail without any changes to the code. This document offers suggestions on how to better detect flaky tests and how to fix or prevent them.

How to detect flaky tests

After writing your test, wrap each describe or it block in a for loop and iterate over it 20-30 times. The number of times you iterate through the loop is arbitrary but the higher the number the better. Having a test pass multiple times should give you confidence that your test is not flaky.

Example:

CODE

for (let i = 0; i < 20; i += 1) {
  describe('the thing', () => {
    it('it does this', () => {
      // commands and assertions go here
    });
  });
}

How to identify the point of failure

Cypress does not give an accurate line number for the point of failure when running tests in headless mode using cy:run. In order to view the exact point at which the test failed, the test needs to be run in headed mode using cy:open. The Cypress browser panel will display a code block with the exact point of failure.

Example:

CODE

src/applications/coronavirus-vaccination/tests/e2e/signup.cypress.spec.js:18:29
  16 |       // Intro page
  17 |       cy.axeCheck();
> 18 |       cy.get('.vads-l-row').contains(
     |                             ^
  19 |         'Stay informed about getting a COVID-19 vaccine at VA',
  20 |       );
  21 |       // Expand all Accordions with keyboard and test for A11y

Call cy.injectAxe() when a new page loads

If you see an error like this:

CODE

TypeError: Cannot read property 'run' of undefined
      at Context.eval (http://localhost:3001/__cypress/tests?p=src/platform/testing/e2e/cypress/support/index.js:25257:442975)

Chances are you forgot to call cy.injectAxe() before calling cy.axeCheck() after Cypress loads a new page.

The undefined object referenced in the error is axe from the cypress-axe module:

CODE

return a.axe.run(t||a.document,u).then(function(e){var t=e.violations;

This error occurs because every time Cypress loads a new page by calling cy.visit(), cy.wait(@alias) (where @alias references a route), or cy.click() (when Cypress clicks on a element like a search button that loads a new page), the axe-core runtime must first be loaded into the page by calling cy.inject().

The cypress-axe documentation explains:

This will inject the axe-core runtime into the page under test. You must run this after a call to cy.visit() and before you run the checkA11y command.
You run this command with cy.injectAxe() either in your test, or in a beforeEach, as long as the visit comes first.

However, as mentioned, it appears that this also applies to other calls that load a new page like cy.wait(@alias) or cy.click().

Race conditions

Because Cypress is asynchronous, race conditions can occur when Cypress ‘gets ahead of the DOM’ and, for example, tries to find or click on an element of the page that hasn’t been rendered yet. Errors that result from race conditions are sometimes difficult to track down. Here are a few tips and a list of useful resources:

exist and be-visible

After calling cy.get() on an element, like a button, for example, make sure it actually exists or is visible before calling cy.click() or some other function on it, like so:

CODE

cy.get('button').should('exist');

or:

CODE

cy.get('button').should('be.visible');

cy.click({ waitForAnimations: true })

cy.click({ waitForAnimations: true }) should be used instead of cy.click() when Cypress clicks on an element that is animated, like an accordion.

This option tells Cypress to wait for elements to finish animating before executing the command.

Race Conditions Resources

Help and feedback

Suggest content changes to this page.
Submit new Platform Website content.
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.