E2E Tests in CI

Overview

This page describes how E2E (Cypress) tests execute in the CI pipeline for vets-website — selection, distribution across parallel containers, execution, and in-PR stress testing. For the E2E tech stack and how to write E2E tests, see the E2E tests parent page. For how tests are chosen based on your changes, see Test Selection in CI.

How E2E tests run in CI

E2E execution is split into three decoupled stages:

1. Selection — decides which E2E tests to run based on your changes and writes the list (e2e_tests_to_run.json). See Test Selection in CI.

2. Distribution (tests-prep job) — expands the selected list and splits it across N parallel containers, writing one list file per container.

3. Execution — each container reads its own list and runs those specs in Cypress. The execution step runs whatever it's given; it does no selection or splitting itself.

This separation means each concern is independently understandable: selection logic, distribution logic, and the Cypress run are no longer tangled in one script.

Parallel container distribution

E2E specs are distributed across parallel containers so the suite finishes faster. Distribution happens in the tests-prep job:

• It reads the selected E2E list and expands patterns to a spec list.

• It derives the container count from the number of specs (clamp(ceil(numSpecs / perContainer), 1, maxContainers); defaults are 6 specs per container, capped at 8 containers, and are environment-overridable).

• It splits the specs into per-container list files (e2e_tests_container_{i}.json).

• Each container in the test matrix downloads its list file and runs exactly those specs.

A runtime integrity guard verifies the split is lossless — the union of all container lists equals the selected list, with no dropped, duplicated, or extra specs — and fails the job if that invariant is violated. This guarantee is built into the distribution step itself.

The distribution strategy is currently count-based (even number of specs per container). The architecture supports a future time-based strategy (balancing by historical run time) without changing the per-container file format or the execution step — only the distribution logic would change. Time-based distribution is deferred until historical timing data is available.

In-PR stress testing (retained)

When your PR adds or changes an E2E spec (*.cypress.spec.js(x)), CI stress tests that spec — running it repeatedly to confirm it passes consistently before it's introduced. This runs on feature branches only and is the stability gate for new E2E tests.

The overnight E2E stress-test workflow (e2e-stress-test.yml) has been removed. Stability validation now happens in-PR, where it's actionable, rather than on an overnight schedule.

Allow list removed

The E2E flaky-test allow list (which filtered known-flaky specs out of a run before execution) has been removed, along with its external data dependency. Previously filtered specs now run. As with unit tests, in-PR stress testing plus more-targeted selection is what keeps flaky exposure in check. If a previously filtered spec is genuinely flaky, treat a new failure as a real stability signal.

Retries and debugging

• Retries: In CI, a failing spec is retried up to twice (retries.runMode = 2) before it's reported as failed.

• Video: Runs are recorded; videos are archived on failure.

• Screenshots: Captured and archived on failure.

• Reports: Mochawesome JSON is produced per container and merged into a single result set.

Use the archived video and screenshots from a failed run to reproduce and diagnose the failure locally.

What you need to do

Nothing changes in how you write or run E2E tests. Keep specs named *.cypress.spec.js(x) and under a tests/ directory so selection and the spec pattern pick them up. If you believe an E2E failure is caused by the CI execution model itself — distribution, the container environment, or the test server — rather than your spec, contact the Platform SRE team via the channel in the change announcement.