Developer docs
Developer docs
Breadcrumbs

Test Selection in CI

Overview

When you open a pull request, CI does not run every test in vets-website. It selects the tests that are relevant to the files you changed. This page documents exactly how that selection works, so you can predict which tests will run for a given change — and know what to do if the selection looks wrong.

Test selection is rules-based: a fixed, documented set of rules maps the files you changed to the tests that run. This replaces the older ad-hoc selection logic, which was undocumented and hard to reason about.

The core principle: run the right tests for what changed — no more, no less.

Which tests run for my change?

Find the row that matches what you touched. More than one row can apply to a single PR — the tests from every matching rule are combined.

I changed…

CI runs…

Code in a single app (src/applications/<app>/**)

That app's unit tests and E2E tests

A platform/shared module (src/platform/<module>/**)

That module's unit tests, plus the unit and E2E tests of every app that imports from the module

App code that shared code imports from

The tests for that shared code

package.json or yarn.lock

Unit and E2E tests for apps that directly import the changed npm packages

Only Markdown (.md) files

Nothing

Only featureFlagNames.json

The 3 feature-toggle machinery specs (not all consumer apps)

.github/workflows/** or /script/**

The /script/ tests

A new or changed unit or E2E spec file

That spec is additionally stress tested (run repeatedly) on your PR to check stability

A mix of app and non-app files

Tests for the changed apps + tests for the changed shared code — there is no all-or-nothing fallback to the full suite

To run everything on demand, use the manual full-suite workflow (workflow_dispatch with RUN_FULL_SUITE).

The selection rules

The table below is the complete, authoritative rule set. Each rule maps a kind of change to the tests it triggers.

Application code changes

#

If this changes…

Run these tests

1

src/applications/<app>/**

<app> unit tests

2

src/applications/<app>/**

<app> E2E tests

Platform / shared code changes

#

If this changes…

Run these tests

3

src/platform/<module>/**

<module> unit tests

4

src/platform/<module>/**

Unit tests for every app that imports from <module>

5

src/platform/<module>/**

E2E tests for every app that imports from <module>

Rules 4 and 5 are the most significant change from prior behavior: a change to shared code now runs the tests of the apps that depend on it, so a regression in a consumer app is caught before merge.

Reverse dependencies (shared code importing from apps)

#

If this changes…

Run these tests

6

src/applications/<app>/** where shared code imports from <app>

That shared code's tests

This generalizes the old "always run static-pages tests" behavior. Instead of unconditionally running site-wide tests, CI detects the actual dependency relationship and runs those tests only when their imports are affected.

Dependency updates

#

If this changes…

Run these tests

7

package.json or yarn.lock

Identify which npm packages changed, find apps directly importing those packages, run those apps' unit and E2E tests

Transitive dependencies (an app imports a platform module that imports the changed package) are covered by rules 4/5 when the platform module has test coverage.

Non-code files

#

If this changes…

Run these tests

8

Only .md files

None

9

Only .github/workflows/**

/script/ tests

10

Only featureFlagNames.json

Feature-toggle machinery tests only (3 specs in src/platform/utilities/tests/feature-toggles/) — not all consumer apps

Rule 10 rationale: adding a flag name to the registry doesn't change consumer app behavior. Consumer apps are covered when their code changes alongside the flag in the same PR (rules 1/2).

Scripts and CI tooling

#

If this changes…

Run these tests

11

/script/**

/script/ tests

Stress testing

#

Condition

Action

12

A *.unit.spec.js(x) file is changed or newly added

Stress test that spec

13

A *.cypress.spec.js(x) file is changed or newly added

Stress test that spec

Stress testing runs a new or changed spec repeatedly on your PR to surface flakiness before it merges. It runs on feature branches only.

Full suite

#

Condition

Action

14

RUN_FULL_SUITE is true (manual dispatch)

Run all unit and/or all E2E tests

Mixed changes (no separate rule)

A "mixed" PR is simply one whose changed files fall into more than one of the categories above — for example an app change plus a platform change plus a dependency bump. There is no dedicated rule for this. Each rule is evaluated independently, and the tests from every rule that matches are combined (unioned and de-duplicated). A mixed PR runs the union of everything its individual rules select; no rule caps or overrides another.

What changed: the old scripts had no way to work out which tests a mixed change actually needed, so any change that didn't match a known pattern fell back to running the entire suite. Rules-based selection replaces that guesswork with targeted per-category selection, so that all-or-nothing full-suite fallback is gone — a mixed PR now runs only the tests its changes map to. The full suite runs only when you request it explicitly (rule 14).

What changed from the previous behavior

Previously

Now

Why

Platform code changes didn't trigger consumer app tests

Platform changes trigger tests for apps that import the changed module (rules 4/5)

Prevents regressions in consumer apps from shared-code changes

Static-pages tests always ran on every changed-files build

Static-pages tests run only when their actual imports are affected (rule 6)

Removes unnecessary runs while preserving the safety net

featureFlagNames.json was silently excluded from selection

Flag changes trigger the 3 feature-toggle machinery tests (rule 10)

Validates the toggle system without a near-full-suite run

.github/workflows/** and /script/** changes triggered no tests

They now trigger /script/ tests (rules 9, 11)

Validates the CI tooling used by workflows

Any non-allowlisted file change caused a full-suite fallback

CI runs only what's relevant to the change (see Mixed changes above)

Reduces CI time for mixed PRs

Dependency updates triggered no targeted tests

Changed packages trigger tests for direct consumers (rule 7)

Catches breakage from dependency updates

Hardcoded special cases (array-builder, forms-config validator)

Covered generically by rules 4/5

Removes one-off rules

Selection logic duplicated across multiple scripts

A single rule set applied consistently for both unit and E2E

Eliminates bugs from divergent logic

The net effect: some PRs run more tests (a shared-code change now runs its consumers), and some run fewer (a mixed PR no longer falls back to the whole suite).

Reporting a selection that looks wrong

There is nothing you need to configure or change in your workflow for rules-based selection — it applies automatically. The one thing to watch for and report is a mismatch:

  • CI ran tests that seem unrelated to what you changed, or

  • CI skipped tests you expected to run for your change.

If you see either, let the Platform SRE team know — post in the questions channel referenced in the change announcement, or open an issue against the CI pipeline. Include the PR link and which tests you expected vs. what ran. Each CI run logs a selection reason (which rules fired and why), which helps diagnose the mismatch.

Surfacing these cases is how the rule set gets refined — a mismatch is a rule gap to fix, not something you should work around.