Planning unmoderated studies

Unmoderated research usually has participants complete simple tasks (as with card sort or tree test) or answer questions (as with a survey). This can help gather feedback from more people more quickly and thereby add to the statistical significance of a study. However, since we are researching for the federal government, we’re required to conduct our studies with extra care like adhering to the Paperwork Reduction Act (PRA), which regulates how the government can collect information from the public. Unmoderated studies also require specific set up and recruiting guidelines, discussed in detail on this page.

Following the Paperwork Reduction Act (PRA) in your research

PRA requires the government, and those working on behalf of the government, to go through a prescribed, months-long process including OMB approval, before asking a standard set of questions to a broad number of people.

Moderated usability research is exempt from PRA because it is direct observation (e.g, usability testing, interviews, co-design) and asking non-standard questions (e.g., not a survey with an identical protocol for each participant).

Unmoderated exemptions from PRA:

• adhere to the “fewer than 10” participant rule

  • split your study into cohorts, each including 9 or fewer participants

  • cohorts should be meaningfully distinct (e.g., Cohort A: users enrolled in health care benefits, Cohort B: users enrolled in GI bill benefits)

• include a separate study link for each cohort

Research tools and participant privacy

Researching for VA requires you to use tools or services that are under a paid license. Free tools collect user data. Should a research participant share PII or PHI while using a free service, it puts our research practice at risk.

An Optimal Workshop account is available for anyone working on VA.gov to use. You can use any available tool, including surveys, card sorting, or tree testing.

Learn how to access the VA.gov team Optimal Workshop account.

Setting up an unmoderated study

1. Read through the PRA primer documentation

2. Create a research plan with meaningfully distinct cohorts of 9 or fewer participants per cohort

   1. Generally it’s easiest to call cohorts by letter designation (e.g., cohort A, cohort B, and so on)

3. Create the study implement (survey, card sort or tree test) in Optimal Workshop

4. At the beginning of the activity, ask for the unique identifier that Perigean gives to participants

   1. Use a “participant identification” field in the “Questionnaire section (not a free form question)

      1. select type “other”

      2. Enter a prompt in the “identifier name” field, like “Please enter the unique code Perigean provided you”

         

5. Generate a separate link to the study in Optimal Workshop for each recruitment cohort

   1. In OptimalSort, go to the Setup tab and then Settings

   2. Under Study setting, find Study link

      1. First, copy the study URL from the setting tab

      2. Then add a different tag to the end of the study URL to create a unique link for each cohort

      3. Link example 1: https://accountname.optimalworkshop.com/toolname/surveyname?tag=A

      4. Link example 2: https://accountname.optimalworkshop.com/toolname/surveyname?tag=B

         

6. Add the study links to corresponding cohorts in the research plan

7. Pilot the study, including the instructions and activity - especially make sure cohort and participant ID come through as expected.

Recruiting for unmoderated studies

1. First, submit your research proposal to the Research Review touchpoint

2. Once your research is approved and ready, Perigean can begin recruitment

   1. Perigean will send out their recruitment email to veterans to gauge interest

   2. Veterans respond to the email

   3. Perigean sends veterans an instructional email, which includes the cohort study link and a unique participant code

3. Veteran participant completes the study and provides the code in the questionnaire

4. Researcher logs into Optimal Workshop and sends codes to Perigean at the end of each day to gauge participation

   1. It’s important to only let Perigean know the code for participants who substantially participated

   2. If a participant abandoned the task, entered their code but didn’t answer any questions, ask Perigean to follow up with them

5. Perigean continues recruitment as needed

6. Let Perigean know when when you’ve closed the study

Tips for conducting unmoderated studies

• Plan ahead to optimize time

  • Leave the sort open for 14 days if possible (one sprint) and use the time to work on other initiatives while the study is open

• Recruit for double the number of participants you want

  • Abandonment is much higher for unmoderated research as compared to moderated studies

• Communicate regularly with Perigean to measure participation and continue recruitment as needed

• It’s still important to pilot test unmoderated studies!

  • Use a fake participant code and make sure all parts of the study work correctly

• Use the Maximum Variation Sampling (MVS) method to make sure your study is inclusive

• Note that some types of studies in Optimal Workshop are not accessible to users with screen readers (e.g. card sort, tree test and first click testing)

• Write thorough instructions for the study

  • For example, many participants have said OptimalSort instructions were difficult to understand

  • Consider a hybrid sort, if applicable, in order to give participants a better understanding of what to do

• When conducting an unmoderated card sort, don’t create more than 30 cards

  • Sorting that many cards requires a high cognitive load and a lot of time, meaning increased abandonment of the exercise

• Do not require participants to go to more than one link to complete the activity

  • This study design decreases participation, increases abandonment

Help and feedback

• Suggest content changes to this page.

• Submit new Platform Website content.

• Get help from the Platform Support Team in Slack.

• Submit a feature idea to the Platform.