Research checklist

This checklist will guide you through all the tasks associated with conducting research on VFS products.

Note: This document contains links to Google Slides. Google Slides are no longer accessible within the VA network. The Platform Content Team is currently working on replacing Google Slides with more accessible documents.

1. Planning for research

3 - 4 weeks before research begins

Planning your study

• Prepare your prototype or mock-up
  Have the thing you’re testing ready! The Collaboration Cycle should conduct a Midpoint Review a week (minimum) before you start your research. They make sure it meets VA.gov quality standards before putting it in front of veterans.
• Make your research inclusive and accessible
  Consider ways to make your research more inclusive and accessible. Some ideas are to set inclusivity requests for recruitment, like using the MVS sampling method; include mobile or have a separate study for mobile usability testing; and even if you won’t have a coded version, there are still assistive technology users you can recruit to test your design. Learn more about conducting research with assistive technology users.

Writing your research plan

TEMPLATE: research plan

• Make a research folder within the Product folder
  Create a research folder (If you don’t currently have one) in your product's folder on GitHub (Add file > create new file > product/research/README.md). The “README.md” does not need to contain any information.
• Make a research folder for your initiative-specific research
  Create an initiative-specific research folder (naming convention: YYYY-MM-Research-Name) within the research folder. Create a markdown file named research-plan.md and save it in the initiative-specific research folder you created. Then, copy and paste from the research plan template into your new file. Fill in your plan using the template; adjust as needed. Store your conversation guide and research plan in this folder. Remember to discuss your research plan with your team.

Research Folder Example:

Product-One (folder) >
Research (folder) >

YYYY-MM-Research-Initiative-One (folder) >

Research Plan (document)

Conversation Guide (document)

Static Screenshots (Sketch file)

Research Findings (document)

YYYY-MM-Research-Initiative-Two >

YYYY-MM-Research-Initiative-Three >

*For additional examples: Combined Debt Portal Research Studies, Public Websites

• Write screener questions
  To ensure you get the right participants for your study, we encourage you to write screener questions along with qualifying responses for the MOST IMPORTANT criteria you want your participants to have, even if you’ve already outlined them in the recruitment criteria table in your research plan.
  
  Perigean includes your screener questions when they sign up and schedule participants using the third-party app, Calendly. Calendly only allows up to seven (7) screener questions. Therefore, Perigean will combine screener questions if you have more than seven (7). Take a look at some Successful screener questions to get started.
• Request a kick-off call with Perigean
  This is an option that Perigean offers so that you can read over both the screener questions and the email before that email is sent out. To request a kick-off call, add the requested date/time in the Recruitment criteria section of the research plan and allow for extra time so that there are 5 business days after the call for recruiting.

Writing your conversation guide

TEMPLATE: conversation guide

• Create a conversation guide
  First, create a markdown file named conversation-guide.md and save it in your research folder (i.e. product-name/research/conversation-guide.md). Then, copy and paste from the conversation guide template into your new file, and adjust the template as needed. Work with your team to complete the conversation guide.
  

2. Research review

At least 10 days before research begins

• Add your study to the repo board
  Create a new ticket using the research repo ticket template.
• Submit your research for review
  See instructions on how to get your research approved and kick off participant recruitment.
• You may also choose to consult on the accessibility of your planned research
  While not required, you may reach out to VA.gov accessibility specialists, #accessibility-help on Slack, for advice on effective and ethical research design for disabilities and assistive technology.

3. Participant recruitment

At least 7 days before research begins

• Initiate your recruitment plan
  Once your research is approved, move your Research Repo card to the “Current” column. Then, if you are using Perigean for recruiting, Shane Strassberg will send the kickoff email signaling Perigean to begin recruiting.
• Kick-off call with Perigean
  Perigean will share their screen with the researcher so you can review the survey and email content that will be sent to participants. Any subsequent changes to screener questions or email must be updated in the research plan.

4. Run a pilot session

Anytime before your first research session

• Identify your pilot participant(s)
  Identify who will participate in your pilot session. We recommend recruiting a developer from your team to get them involved early in the process.
• Set up the pilot session
  Set a meeting time with your pilot. Include the guidelines about participating in a pilot with the invite so folks know what to expect.
• Run the pilot session
  Run the pilot session and note where any adjustments could make future sessions smoother.
• Make any adjustments to your conversation guide
  Ensure note-taker(s) and moderator(s) have the most recent version before the sessions begin.

5. Conduct sessions

Before research sessions

As Perigean schedules sessions

• Assign and confirm notetaker(s)
  This should include a member of your team but could include others. Ask your notetaker(s) to review guidance on taking notes. If you can’t find someone to take notes, Perigean is available to assist. Learn how to prepare for Perigean to take notes.
• Prepare for note-taking
  Create a “session-notes” folder within your research study Github folder and store each session as a .md document. Copy the conversation guide over to your .md file before each session so that you have a structured way to capture your notes.
• Invite observers
  Invite your team and stakeholders to sessions. Limit your observers to 3-4 per session so participants aren’t overwhelmed. You should empower participants by allowing them to decide if they are comfortable having observers. Ask observers to review information about observer guidelines and attendance etiquette.
• Prepare as moderator
  Be familiar with the conversation guide, study implement(s), and prototype(s). Review tips on moderating, guidelines for recording sessions,and guidance on participant and researcher safety and exit strategies.

During research sessions

• Start a thread in #feedback-backchannel (ex: Dependency Verification Usability P1 starts in 10 minutes :thread: )
• Type “observer instructions” in the channel at the beginning of your day of sessions to give them tips
• Make sure you mute all notifications, and close unnecessary applications
• Make sure all attendees (except participants) are muted
• Have your conversation guide somewhere accessible that the participant cannot see
• Make sure you have transcriptions enabled
• Make sure to hit record after permission is granted!
• Keep handy the Tech set-up checklist and Troubleshooting Research Sessions in case the participant needs help during the session.
• Refer to tips on moderating and responding to emergencies
• At the end of your session, take a look at the #feedback-backchannel and see if anyone brought up questions that you can address with your participant
• You may choose to take quick notes, but your primary duty as moderator is to run a smooth session. That’s why you have a notetaker!

After research sessions

• Check-in with notetakers and observers to capture their top 3 observations. Tip: ask them to put these in the session #feedback-backchannel thread or a Mural you created for this!
• Have a daily or weekly debrief with your team to help digest sessions
• Summarize what you’ve heard by writing up key findings/themes. You may use the Topline Summary template.
• Share this summary with the team and observers to see if they feel anything was missing based on their observations
• Download any recordings you need, but only keep them for as long as necessary to complete synthesis, then securely dispose of them. You’ll have up to seven days to download an individual recording from the day it was recorded while the study is ongoing and up to 24 hrs after the close of the study. Perigean will confirm that you are finished downloading your recordings before deleting them. Make sure to review and ensure you understand the guidelines for recording user research
  Do not upload any original session recordings to the cloud (Github, Dropbox, Google Docs, etc.) or any public or private server. The risk of accidentally disclosing PII outweighs any benefit provided by sharing original session recordings.

6. Synthesize data

Use whatever method and tool that works for you and your team. The key things you’ll want to get out of your research to write up your research findings are:

• 5-10 top findings/themes, along with supporting quotes
• Determine whether you proved your hypotheses true or false
• Come up with recommendations based on your findings, including supporting evidence.

7. Share your findings with colleagues

• Write up your research findings
  Use the research findings template to document your findings. This should be stored in your GitHub research folder. If you’d like to include a video clip of a participant, make sure to scrub all PII and PHI. Remember VFS work is about the digital experience, so that should be the focus of your findings since that’s what we can act on.
• Update the research repo board
  Follow these guidelines for what to do when your research is complete.
• Share your findings with other colleagues
  Share your research with your team and stakeholders, and others. If the template above doesn’t fit some audiences well, you may use this VA presentation template.
• Share with the broader design practice
  Reach out to Naomi Marcussen on Slack to sign up to share your findings with others at the weekly Design + Content + Research meeting. Use this template to create your presentation.
• Contribute to the Design System
  Does your research have design system implications? Find out how to contribute to the design system.

Help and feedback

• Suggest content changes to this page.

• Submit new Platform Website content.

• Get help from the Platform Support Team in Slack.

• Submit a feature idea to the Platform.