Planning unmoderated studies

Unmoderated research usually has participants complete simple tasks (as with card sort or tree test) or answer questions (as with a survey). This can help gather feedback from more people more quickly and thereby add to the statistical significance of a study. However, since we are researching for the federal government, we’re required to conduct our studies with extra care like adhering to the Paperwork Reduction Act (PRA), which regulates how the government can collect information from the public. Unmoderated studies also require specific set up and recruiting guidelines, discussed in detail on this page.

Following the Paperwork Reduction Act (PRA) in your research

PRA requires the government, and those working on behalf of the government, to go through a prescribed, months-long process including OMB approval, before asking a standard set of questions to a broad number of people.

Moderated usability research is exempt from PRA because it is direct observation (e.g, usability testing, interviews, co-design) and asking non-standard questions (e.g., not a survey with an identical protocol for each participant).

Unmoderated exemptions from PRA:

• adhere to the “fewer than 10” participant rule

  • split your study into cohorts, each including 9 or fewer participants

  • cohorts should be meaningfully distinct (e.g., Cohort A: users enrolled in health care benefits, Cohort B: users enrolled in GI bill benefits)

• include a separate study link for each cohort

Research tools and participant privacy

• Researching for VA requires you only use tools or services that are under a paid license - that license may be paid for by either VA or a contractor team.

• Free tools collect user data. Should a research participant share PII or PHI while using a free service, it puts our research practice at risk.

Setting up an unmoderated study

1. Read through the PRA primer documentation

2. Create a research plan with meaningfully distinct cohorts of 9 or fewer participants per cohort

   1. Generally it’s easiest to call cohorts by letter designation (e.g., cohort A, cohort B, and so on)

3. Create the study implement (the card sort or tree test) in Optimal Workshop

4. At the beginning of the activity, ask for the unique identifier that Perigean gives to participants

   1. Use a “participant identification” field in the “Questionnaire section (not a free form question)

      1. select type “other”

      2. Enter a prompt in the “identifier name” field, like “Please enter the unique code Perigean provided you”

         

5. Generate a separate link to the study in Optimal Workshop for each recruitment cohort

   1. In OptimalSort, go to the Setup tab and then Settings

   2. Under Study setting, find Study link

      1. First, copy the study URL from the setting tab

      2. Then add a different token to the end of the study URL to create a unique link for each cohort

      3. Link example 1: https://accountname.optimalworkshop.com/toolname/surveyname?token=A

      4. Link example 2: https://accountname.optimalworkshop.com/toolname/surveyname?token=B

         

6. Add the study links to corresponding cohorts in the research plan

7. Pilot the study, including the instructions and activity - especially make sure cohort and participant ID come through as expected.

Recruiting for unmoderated studies

1. First, submit your research proposal to the Research Review touchpoint

2. Once your research is approved and ready, Perigean can begin recruitment

   1. Perigean will send out their recruitment email to veterans to gauge interest

   2. Veterans respond to the email

   3. Perigean sends veterans an instructional email, which includes the cohort study link and a unique participant code

3. Veteran participant completes the study and provides the code in the questionnaire

4. Researcher logs into Optimal Workshop and sends codes to Perigean at the end of each day to gauge participation

   1. It’s important to only let Perigean know the code for participants who substantially participated

   2. If a participant abandoned the task, entered their code but didn’t answer any questions, ask Perigean to follow up with them

5. Perigean continues recruitment as needed

6. Let Perigean know when when you’ve closed the study

Tips for conducting unmoderated studies

• Plan ahead to optimize time

  • Leave the sort open for 14 days if possible (one sprint) and use the time to work on other initiatives while the study is open

• Recruit for double the number of participants you want

  • Abandonment is much higher for unmoderated research

• Communicate regularly with Perigean to measure participation and continue recruitment as needed

• It’s still important to pilot test unmoderated studies!

  • Use a fake participant code and make sure all parts of the study work correctly

• Use the participant roster to check inclusivity, and fill out the recruitment checker

• Note that Optimal Workshop is not accessible to users with screen readers

• Write thorough instructions for the study

  • For example, many participants have said OptimalSort instructions were difficult to understand

  • Consider a hybrid sort, if applicable, in order to give participants a better understanding of what to do

• When conducting an unmoderated card sort, don’t create more than 30 cards

  • Sorting that many cards requires a high cognitive load and a lot of time, meaning increased abandonment of the exercise

• Do not attempt a multi-party unmoderated study

  • This study design decreases participation, increases abandonment

• Do not require participants to go to more than one link to complete the activity

  • This study design decreases participation, increases abandonment

---

Help and feedback

• Get help from the Platform

• Create an issue ticket to suggest changes to this page

• Submit a feature request