E2E Tests in CI

Last Updated: August 12, 2026

Overview

This page describes how E2E (Cypress) tests execute in the CI pipeline for vets-website — selection, distribution across parallel containers, execution, and in-PR stress testing. For the E2E tech stack and how to write E2E tests, see the E2E tests parent page. For how tests are chosen based on your changes, see Test Selection in CI.

How E2E tests run in CI

E2E execution is split into three decoupled stages:

1. Selection — decides which E2E tests to run based on your changes and writes the list (e2e_tests_to_run.json). See Test Selection in CI.

2. Distribution (tests-prep job) — expands the selected list and splits it across N parallel containers, writing one list file per container.

3. Execution — each container reads its own list and runs those specs in Cypress. The execution step runs whatever it's given; it does no selection or splitting itself.

This separation means each concern is independently understandable: selection logic, distribution logic, and the Cypress run are no longer tangled in one script.

Because selection maps changed files to specs and the build is entry-filtered (only changed apps are scaffolded), a selected platform or site-wide spec can cy.visit() an app that isn't in the build and hit a 404. That is a spec-authoring constraint, not an execution-model bug — platform specs must only visit /. See Writing an end-to-end test and the entry-filtered-build note in Test Selection in CI.

Parallel container distribution

E2E specs are distributed across parallel containers so the suite finishes faster. Distribution happens in the tests-prep job:

• It reads the selected E2E list and expands patterns to a spec list.

• It derives the container count from the number of specs (clamp(ceil(numSpecs / perContainer), 1, maxContainers); defaults are 6 specs per container, capped at 8 containers, and are environment-overridable).

• It splits the specs into per-container list files (e2e_tests_container_{i}.json).

• Each container in the test matrix downloads its list file and runs exactly those specs.

A runtime integrity guard verifies the split is lossless — the union of all container lists equals the selected list, with no dropped, duplicated, or extra specs — and fails the job if that invariant is violated. This guarantee is built into the distribution step itself.

When selection produces patterns but they expand to zero specs, the distributor decides whether that is expected or a problem before emitting containers. It checks whether E2E specs exist anywhere under each selected pattern's static base path (the literal prefix before the first glob segment) — not merely whether that base folder exists. A base is treated as stale, and the job fails loudly, when it no longer exists on disk (a renamed or typo'd app folder) or it exists but still contains E2E specs the precise selection failed to match (a moved/renamed tests/ directory, or a broken full-suite glob). If no targeted base is stale, the app simply has no E2E tests: the job emits zero containers and skips cleanly. This distinguishes an app that legitimately has no E2E tests (clean skip) from a selection that would silently drop real coverage (hard failure) — the latter is the hazard the guard exists to catch.

The distribution strategy is currently count-based (even number of specs per container). The architecture supports a future time-based strategy (balancing by historical run time) without changing the per-container file format or the execution step — only the distribution logic would change. Time-based distribution is deferred until historical timing data is available.

In-PR stress testing (retained)

When your PR adds or changes an E2E spec (*.cypress.spec.js(x)), CI stress tests that spec — running it repeatedly to confirm it passes consistently before it's introduced. This runs on feature branches only and is the stability gate for new E2E tests.

The overnight E2E stress-test workflow (e2e-stress-test.yml) has been removed. Stability validation now happens in-PR, where it's actionable, rather than on an overnight schedule.

Allow list removed

The E2E flaky-test allow list (which filtered known-flaky specs out of a run before execution) has been removed, along with its external data dependency. Previously filtered specs now run. As with unit tests, in-PR stress testing plus more-targeted selection is what keeps flaky exposure in check. If a previously filtered spec is genuinely flaky, treat a new failure as a real stability signal.

Retries and debugging

• Retries: In CI, a failing spec is retried up to twice (retries.runMode = 2) before it's reported as failed.

• Video: Runs are recorded; videos are archived on failure.

• Screenshots: Captured and archived on failure.

• Reports: Mochawesome JSON is produced per container and merged into a single result set.

Use the archived video and screenshots from a failed run to reproduce and diagnose the failure locally.

What you need to do

Nothing changes in how you write or run E2E tests. Keep specs named *.cypress.spec.js(x) and under a tests/ directory so selection and the spec pattern pick them up. If you believe an E2E failure is caused by the CI execution model itself — distribution, the container environment, or the test server — rather than your spec, contact the Platform SRE team via the channel in the change announcement. One exception that is a spec issue: a platform or site-wide spec (src/platform/**) that cy.visit()s a specific app route can 404 under the entry-filtered build — such specs must only visit / (enforced by ESLint). See Writing an end-to-end test.