The purpose of this document is to co-create a single, cross-discipline, cross-contract, VA-wide accessibility strategy document we can point to as a north star to answer:

  • How we define accessibility for VA.gov

  • How we’ll know we’re successful (or not), and

  • How we’ll measure and keep track of our progress.

How did we make this? 

We started this effort through a prioritization exercise based on problems our accessibility specialists identified in our a11y specialist problems document (access restricted to accessibility specialists). 

What does this cover?

For now, this strategy document covers VA.gov (specifically the parts that the Office of the Chief Technology Officer [OCTO] is responsible for), but we’re looking to extend this to the native mobile app and beyond in the future. 

How will this change? 

Everything in this strategy is intended to be flexible and may change over time. We would love to have your feedback on our #accessibility-ops channel.

Mission

Empower product teams to collaborate with disabled veterans and caregivers by enhancing the culture and tools used to co-create and maintain services that are accessible beyond compliance.  

Vision

Every disabled Veteran and caregiver has guaranteed access to equitable, easy to use, self-service tools without needing to request accommodations1.

Key Success Metrics

We plan to review the metrics below on a monthly basis using our VA A11y Dashboard (Google Sheets). Changes in metrics should be reviewed by accessibility leads to evaluate effectiveness of tactics, inform new tactics, create new conversations, or re-evaluate broader accessibility strategy. 

None of these key metrics are permanent and are meant to be discussed and modified over time by the team.

Percentage of research studies with the accessibility label

Helps us measure the inclusion of disabled Veterans in our research.

  • Why? Most VA.gov products have not been tested with disabled Veterans who use assistive technology or have compounded disabilities.

  • How will we track this?

    • We are using the accessibility label, accessibility, in the research repo. Currently, the criteria for the accessibility label is up to the researcher. It's most often applied to research specifically looking at accessibility concerns or when the majority of participants use assistive technology (AT). It's also sometimes being applied when research includes any AT user participants.

    • In the future, we plan on collaborating with the research ops team to  either (a) automate inputs for requested assistive technology users or (b) manually track how many Veterans emerge in studies through the recruitment checker or by following studies in the research ops board.

  • How will we monitor and use it?

    • If the number consistently dips over time, it’s a sign we’re not including Veterans with disabilities and that we should take a closer look at studies.

    • This metric can help open up conversations related to blockers around Veterans with disabilities who aren’t able to participate in research, for example, inaccessible prototypes.

  • Submetrics

    • In the future we may be able to use the recruitment checker to keep track of what types of assistive tech are being tested with specifically. Work with research ops to standardize how we’re tracking these

Percentage of staging reviews with no major issues (defects 0-2) written

Helps us measure at large if accessibility is succeeding at shifting left (regardless of what tactics are being used).

  • Why? Some Collaboration Cycle reviews still produce major defects.

  • How will we track this?

  • How will we monitor and use it?

    • We can add this to our accessibility dashboard and start checking in on it every quarter to see if the percentage is increasing or decreasing over time. In the future when this metric is a little more stable, we may be able to set up alerts.

    • If this number increases, it’s a signal that our tactics for shifting left are working. 

    • If this number decreases, it can open conversations around exploring different tactics for shifting left. For example, we might ask “are there any patterns among the issues being detected by type and team?” We are interested to see if there is a notable difference between teams with and without embedded accessibility specialists.

Number of unique VFS teams that contact accessibility specialists

Helps us measure how proactive or involved VFS teams are in collaborating with accessibility specialists early and often. 

  • Why? In an ideal world, VFS teams should be more incentivized to learn and seek out effective accessibility practices from accessibility specialists.

  • How will we track this?

    • As an MVP, we’ll track unique questions asked using this spreadsheet and keep a manual tally of teams that have reached out to us each month. 

    • In the future, we hope to repurpose our questions spreadsheet into a method of identifying common questions and providing quick responses (if something has been asked before, e.g. a self-service knowledge repository).

  • How will we monitor and use it?

    • If the number of unique questions decreases significantly, we'll be motivated to look into the causes e.g. is it motivation based? Are teams aware that office hours exist? Etc. The spreadsheet also allows us to look at the sub-metrics related to what teams are asking questions, where they’re asking questions, and what kinds of questions they’re asking.

Percentage of accessibility specialists with a balanced workload

Helps us measure if accessibility specialists are: working sustainably, not suffering from “burnout”, and feel satisfied with their work.

  • Why? Accessibility specialists burn out quickly and have been historically overworked. 

  • How will we track this?:

    • Weekly, on a Google spreadsheet, as part of our weekly syncs.

  • How will we monitor and use it?:

    • Our accessibility leads intend to use this to keep track of specialist workloads and know when to step in to help and mitigate burnout before it happens.

    • As an accessibility practice, this can help inform decisions to assign or reassign tasks for VA.gov wide accessibility efforts.

    • Support hiring decisions

    • In the future we may want to consider adding on to this by using a count of backlog tickets and time to completion.

[ 🛠️ In Progress] Average time to close out accessibility bugs 

Helps us measure if accessibility is being prioritized by teams.

  • Why? Many accessibility issue tickets aren’t prioritized and don’t get fixed in a timely manner. We have 500+ issues that remain unresolved.

  • How will we track this?:

    • We’re working with the Platform Analytics team on building Domo dashboards around issue ticket lifespans with the intention of being able to filter by accessibility. 

  • How will we monitor and use it?:

    • When established, we can use this metric over time to start conversations on which teams are (or aren’t) prioritizing accessibility and why.

[ 🛠️ In Progress] Percentage of components and patterns we consider to have deficient (or sufficient) accessibility guidance

  • Why? This will help us understand how embedded accessibility is in the design system which scales across to all VFS teams.

  • How will we track this?:

  • How will we monitor and use it?:

    • If the metric remains stagnant or trends in a negative direction, it’s a possible sign that many new inaccessible components are being added or the remediation of existing elements isn’t being prioritized.

    • In the future, we may also want to expand this to tracking severity of issues within the design system. That depends on if there will be a consistent review to components (e.g. the number of tickets, and their severity, will fluctuate more based on the consistency and spread of audits and less on the actual number of issues present that have yet to be discovered).

Strategies

These broader strategies will help us form tactics which we’ll evaluate based on the key metrics listed above. We’ll select tactics to experiment with and add them to our backlog in our VA A11y Dashboard.

Some strategies listed here are intended for a future state of accessibility at the VA and may not be applicable now. For example, we don’t have the ability to collect certain key metrics to evaluate the accessibility of internal tools, methods, and processes.

1. Self-service benefits accessed on VA.gov by disabled Veterans are co-designed with disabled Veterans.

Problem this is meant to solve:

  • We don’t have any standardized method or practice to measure and learn if our accessibility recommendations are having a positive impact on the user experience for disabled Veterans. 

  • While we’re confident that our recommendations can create access (for example, we know folks can physically enter a building because we set up a ramp), a truly inclusive, easy to use, self-service experience is most effectively designed with users (for example, including disabled people into research may unveil that they don’t want a ramp by the back door dumpster, they want a flat entrance without stairs).

  • Mitigates us from creating disability dongles (well-intended, but ineffective solutions for disabled people)

2. VA.gov is the standard for accessibility beyond compliance in both the private and public sector.

Problem this is meant to solve:

  • Aside from (arguably) gov.uk, there aren’t many excellent “standards” for inclusive services for the disabled experience. Referencing subpar, inequitable experiences from the private industry may hurt our efforts more than help them.

  • Aiming to become the new standard instead of just catching up to a low bar can enable us to think even further beyond compliance. We believe this will empower us to focus more on the disabled Veteran experience, not just an automated count of technical issues.

  • Sharing what we learn with other agencies can (long term) help create a more equitable American democracy.

3. All product teams have the needed space, time, knowledge, and power to embed accessibility as part of their key responsibilities.

Problem this is meant to solve:

  • Accessibility resources are currently scattered across VA.gov with little to no maintenance or organization.

  • Some teams don’t know where to get accessibility guidance, who to reach out to, etc. 

  • Some teams may not be aware that accessibility is a part of their key responsibilities when launching products. (They may also be unaware of what those specific responsibilities are.)

  • Accessibility is not (yet) an organized practice. It is currently a loosely organized group of specialists collaborating on Slack.

4. The systems and tools that allow work to happen also enable, not block, accessibility beyond compliance.

Problem this is meant to solve:

  • In order for us to employ disabled employees (who may themselves be Veterans), we need to make internal ops (tools, methods, etc.) and culture more accessible and inclusive to them. This is important as it's both a legal mandate (which Congress wants prioritized) and a critical component in inclusive and participatory design. Hiring more disabled Veterans empowers our teams to have more perspectives centered around lived experience, and ultimately leads to more equitable products and services.

  • Accessibility specialist work is prone to burnout. Without clear expectations on scope, resources, and support, we may end up experiencing high levels of turnover.

  • Existing systems (Forms Library and Design System) are not yet accessible by default. Fixing underlying systems can scale accessibility to new heights.

Contributors

It takes a lot of people with lots of different, diverse perspectives to make this work. Credit go to the following individuals for writing, reviewing, or providing direction and support:

  • Martha Wilkes (our executive sponsor)

  • Angela Fowler

  • Brian DeConinck

  • Tiffany Pender

  • Laura Flannery

  • Brea Blackwelder

  • Courtney Chambers

  • Meli Manak 

  • Jared Cunha

  • Ryan Smith

  • Sarah Koomson

  • Josh Kim

  • David Kennedy

  • Eli Mellen

  • Shawna Hein

  • Mickin Sahni

  • Mike Gifford

  • Jenny Wang

  • Anish Kshatriya

  • Clayton Zook

  • Liz Lantz

  • Matt Dingee

  • Naomi Marcussen

  • Shira Goodman

  • Danielle Thierry

  • Tim Wright

  • Michelle Middaugh

  • Nina Anusavice 

  • Christopher Walker

  • Cory Trimm

Want to become a contributor? Let us know at #accessibility-ops.

Footnote

  1. Accessibility can be thought of as a continuum. Inaccessible experiences exclude people. Accommodations are modifications made for disability on an as-needed basis. Accessible experiences provide equal access to everyone without disability tax, emotional tolls, microaggressions, or psychological burdens.