Researching with mobile users

Many if not most veterans use mobile devices, so we encourage testing designs on mobile and desktop.

  • Users can install Zoom on their phones or use the "laptop hugging method."

  • Perigean can recruit mobile users for you. Be sure to meet with Perigean folks before sessions to make sure everyone understands the process and that Perigean knows how to coach users and help solve technical issues.

  • Be prepared to walk through the technical setup for mobile (or any) testing with a Veteran if they haven't figured out the proper setup before the call begins.

  • Refer to this guide to learn how to use zoom for user research.

Learn more about planning for and conducting mobile research.

Recording your sessions

You may record the audio and your screen for your sessions, however:

  • You must ask the user's permission at the start of the session, and you must let your participant know on the record that you have started the recording. See conversation guide template.

  • Perigean will keep the recordings for up to 1 week following the study. As a lead researcher, it is your responsibility to download any recordings you need for synthesis and destroy the files when you are done.

  • Do NOT post the session recordings to GitHub or Slack.

  • If you want to keep a clip for communication purposes, it is YOUR responsibility as the lead researcher to strip the video clip of PII or PHI.

Communicating during the session

There is a channel in DSVA Slack called #feedback-backchannel. Make sure to be in that channel when running your sessions and that your observers know about it.

  • Type "observer instructions" in the channel at the beginning of your day of sessions to give them tips

  • Observer instructions can also be found here.

  • At the end of your session, take a look at the feedback backchannel and see if anyone brought up questions that you can address with your participant.

Taking notes

Primary notetaker

Each session should have a primary notetaker. Ideally, this person should be a team member but could also be a fellow designer or a content person. The primary notetaker's goal is to take verbatim notes. Do not input your observations/feelings/solutions in the notes. You can do this via Excel, Word, a .md file, Handrail, or any other way that works for you.

File naming for verbatim notes: Participant#.researchstudy notes.date and time (P1.personalizationusability notes.201706041300 = Participant 1, personalization usability study notes, June 4, 2017, at 1 pm.)

You must scrub all notes of all PII before uploading them to Github. PII is all content that could be used to identify the participant. Name, age, date of birth, location or specific VA facility (unless that is critical to the study), diagnoses, etc.

An example of verbatim notes can be found here.

Observers

Before running sessions, it’s nice to provide a means for your team members and/or stakeholders to take notes. Share whatever method you will be using with your observers a day in advance of starting any sessions

You can do this in whatever way works for you, some ideas:

  • Have observers slack their observations in the feedback channel or your team channel and then compile them in your spreadsheet

  • Create a Mural board and share out. A Mural template is available to help observers take notes during your sessions.

Please note, there's a difference between an observation (what a participant said or did) and a recommendation ("We should make Y change...") Notes should be observations and specific quotations that stuck out.

No-shows

Life happens. Participants sometimes don't show up to sessions or have emergencies. Be prepared to need to reschedule some of your sessions during your study. We recommend asking for more participants than you will need and providing ample open slots for research so that you can meet your targets within your timeframe.

Sharing video

There is no need to share your video or ask participants to share their video. Veterans are not instructed to share or expected to share video because of these reasons:

  • logistics - sometimes getting screensharing up and running is hard enough!

  • bandwidth - Veterans in low bandwidth locations may not be able to share video

  • comfort - we don't want to make any user feel uncomfortable

Moderating tips

Veteran specific moderating tips:

  • You will hear hard stories from veterans. It’s OK to sympathize with the participant. Saying something like “I’m so sorry you had to go through that” is perfectly acceptable after hearing a story before moving on to your task or follow-up. It is almost never necessary or relevant to the goals of any VSA or VA.gov study to dig deeply into folks' trauma, however. The best attitude when in doubt is a middle ground where you show compassion but then move on to the next task or question in a graceful way. E.g. "that sounds so hard/frustrating/difficult. [Switch back to script or intention of study] Could you tell me a bit more about your experience applying for that benefit?"

  • If someone asks you a question about the legal rules of how a program works, make sure to say "I'm not sure. But you can go to VA.gov and call the support line listed there." Don't guess or say anything even if you might think you know the answer because we don't want to get into legal difficulties.

  • Avoid saying "Thank you for your service" to Veterans. Some Veterans have a negative association with that phrase.

General moderating tips:

  • Remember not to ask questions that are the same wording as what's on your designs. For example, if you have a "learn more" dropdown don't ask "how would you learn more about this?" It's pretty leading if you use the same terminology. Instead something like "if you're confused, where would you go to get more information?"

  • Ask open-ended questions and avoid Yes/No prompts.

  • Follow-up questions with “Why?” and “Tell me more” if the user has given you a short statement or not provided enough information.

  • Avoid using morale/biased language like "good!" or "perfect!" when a user completes a task in the way you'd expect.

  • If someone asks you how something works or what happens when they click on something, respond with "how would you expect it to work?" rather than answering their question

Participant safety and responding to emergencies during remote research sessions

As researchers, our number one priority is to keep our participants safe. When working with Veterans, it is particularly important to be aware of our participants' psychological and physical well-being during our research sessions. If you find yourself in a session with a Veteran experiencing a medical or psychological emergency, follow the protocol below to request assistance. Read more about our participant safety and emergency protocols.