Planning Unmoderated Studies
Introduction
Conducting research for a government project comes with constraints. We must adhere to the Paperwork Reduction Act (PRA), which regulates how the government can collect information from the public. In general, PRA requires that the government must accept and consider public comments and get formal approval from OMB in order to collect information from the public. This is a months-long process.
Most of the user research we do is considered exempt from PRA because we are conducting direct observation (usability testing, interviews, co-design) and asking non-standard questions (e.g., not a survey with the identical protocol for each participant). Conducting a statistically significant survey with hundreds or thousands of respondents requires PRA approval.
In order to be exempt from PRA when conducting unmoderated testing such as card sorts or tree tests, we must adhere to the "fewer than 10" participant rule. That means that when structuring your study, you must:
create meaningfully distinct cohorts of participant recruitment (e.g. Cohort A: users enrolled in health care benefits, Cohort B: users enrolled in GI bill benefits)
include fewer than 10 participants per cohort
include a separate study link for each cohort
Note on tools
Whether you are doing unmoderated or moderated research, any tool/service you use should be approved by the VA and must be under a paid license -- whether that license is paid for by VA or a contractor team. Using free tools means that the service gets your data. Should a research participant inadvertently share PII or PHI, using an unlicensed service puts our research practice at risk.
How to set up an unmoderated study
Read through the PRA primer documentation
Create [research plan] with meaningfully distinct cohorts of fewer than 10 participants per cohort
Create the card sort/tree test in Optimal Workshop
As part of the card sort/tree test, include a short answer question for participants to provide their unique code (given by Perigean during the recruitment process)
Generate a separate link to the study in Optimal Workshop for each recruitment cohort
Add study links to corresponding cohorts in the research plan
Draft recruitment email that explains what the study is for and what a participant can expect like brief task description and estimated length to complete.
Draft instructions email that gives clear, simple directions for completing the study. Perigean will include the appropriate study link for each cohort and a unique participant code.
Recruitment process
Perigean will send out your recruitment email to Veterans to gauge interest
Veterans respond to the email
Perigean sends the instruction email, including a link to the sort and unique code
Veteran completes the study and provides the code in the questionnaire
Researcher logs into Optimal Workshop and sends codes to Perigean every few days to gauge participation
Perigean sends continues recruitment as needed
Tips and recommendations
Note that Optimal Workshop is not accessible to users with screen readers
Write thorough instructions for the sort
Many participants mentioned that the OptimalSort instructions were difficult to understand
Potentially do a hybrid sort in order to give participants a better understanding of what to do
Don’t create more than 30 cards
The cognitive load for sorting more than 30 cards is heavy and may also result in low esteem from participants and abandonment of the exercise
Do not attempt a multi-party unmoderated study that requires participants to go to more than one link to complete the activity
We have seen decreased participation with this study design
Plan ahead to optimize time
Leave the sort open for 14 days if possible (one sprint) and use the time to work on other initiatives while the study is open
Recruit for double the number of participants you want
Abandonment is much higher for unmoderated research
Communicate regularly with Perigean to measure participation and continue recruitment as needed
Help and feedback
Create an issue ticket to suggest changes to this page