Homework 3: usability test

In this assignment, you will be performing a usability test of a privacy or security interface of your choice.

Due

This homework is due 23:59 on Tuesday, March 26.

Collaboration

This assignment should be completed in groups of 2.

Assignment

Your task is to conduct a usability test of a security or privacy interface. In simple terms, you’ll need to recruit a real person and observe them use the interface, drawing conclusions about any usability issues they encounter.

Below are the steps you’ll need to follow.

Decide the software interface and task you will be studying

You’ll need to choose one concrete task as part of a larger interface. It must follow the same requirements as in the previous assignment but it does not have to be the same interface:

  • It must have something to do with security and privacy.
    • Try to pick something reasonably complex to make it interesting!
    • If in doubt, please ask.
  • There must be at least 4 distinct steps to the task you are studying.
  • It must not be an example we used in class or one of the following tasks:
    • Authenticating with two-factor authentication
      • (However, more complex operations related to MFA may be okay.)
    • Logging in using a password manager or adding a login to one

Prepare your task

Start by identifying the end goal of your task. (Usability studies can be exploratory or oriented towards specific tasks, but, for the purposes of this assignment, having a concrete objective that you can communicate to your user will be easier.)

Then, think about what the starting point is going to be. For example, if creating a user account is not part of your usability test, you might need to assume the account has already been created (and prepare working credentials for your user before the usability test).

Once you have the start and end points, write down the goals in a way that is concrete and precise but does not instruct the user about the specific steps to follow. For example:

Your goal is to log in to the encrypted email provider and send the message Hi! This is a secret message! to recipient@example.com in a way such that only the recipient will be able to read it.

Have these instructions written out and everything else ready to go when you start the test.

Recruit your participant

Usability tests are often conducted with 5 (or more) participants but, for this assignment, I’m only requiring that you conduct a test with one person. This person must be:

  • Not someone else in this course
  • Not familiar with the task or interface you’re trying to test
    • (For example, don’t recruit a network security professional to test Wireshark.)

Reserve 30–60 minutes of your participant’s time to make sure you’re not rushed.

Prepare to moderate the usability test

When you’re running your usability test, you will not be sitting back and just watching the participant — you’ll need to play a more active role to elicit the participant’s thoughts on the interface.

You have a few options about how and when to get feedback, which are summarized well in this article by Jennifer Romano Bergstrom:

  • Concurrent Think Aloud
  • In Retrospective Think Aloud
  • Concurrent Probing
  • Retrospective Probing

For this class, you should have your participant engage in Concurrent Think Aloud technique, meaning that you want to have them narrate everything they’re doing, why they’re doing it, and the thoughts that come to their head in a stream of consciousness manner.

After the test starts, you will most likely need to remind your participant to think aloud more. You can do this with phrases like “uh-huh” and “say more.” If they stop altogether, ask them, “can you tell me what you’re thinking through right now?”

Conduct the test

You can conduct the test remotely, but I would strongly prefer that you conduct it in person.

The test can be conducted on the participant’s own device (preferred, since this is more realistic) or yours, depending on what you find most convenient.

As you prepare to start, provide your participant with the necessary background information and inform them of their task.

Before you begin, explain to your participant the Concurrent Think Aloud protocol (described above). You should also make it clear to your participant that this is a test of the interface, not them and that they should feel comfortable expressing any and all thoughts and critiques that come to mind.

For this assignment, one person in your pair should be the moderator (as described above), and the other should take notes on everything the participant is saying out loud. You should also record the complete screen where the test is taking place. (Audio recording is not necessary.)

You should generally decline to answer any questions the participant may ask you about the interface. If they ask you for help, or pose other questions about the interface, tell them to pretend that you’re not there right now, but that you’d be happy to discuss any of their questions or comments at the end.

If the participant gets stuck, you should generally let them struggle and try to figure the way out on their own. Only offer suggestions if a long time has passed and you feel pretty certain that they won’t be able to continue without your assistance.

Once the participant has completed their task, ask them for more holistic feedback on their experience. What are their thoughts on the interface? What did they find confusing? What are their suggestions for improvement?

Come up with conclusions

Once the usability test is done, you should analyze your data to come up with conclusions. What difficulties did the participant experience? What elements of the interface caused them? (Conversely, what are some things that worked well?) Overall, what are the major themes of your findings?

Based on these, come up with recommendations you would provide to the interface designers. Also think about the next steps you would take as a researcher: for example, are there open questions that were left unanswered by this particular usability test?

Resources

As with all UX research methods, there are many resources online that can help you conduct usability tests, for example from Usability.gov and a usability test checklist and guide to remote tests from the Nielsen Norman Group.

What to turn in

You must submit 3 things:

  1. The screen recording from the usability test (as an MP4)
    • As noted above, audio is not expected
  2. Raw notes taken during the usability test (as a PDF)
  3. Your write-up, described next (as a PDF created using LaTeX)

Your write-up should include the following:

  • Name and link to the software or service you analyzed
  • A description of the task (including its goals) you chose
  • A description of the technical background and any relevant experience of your participant
  • A description of the major themes and findings from your usability test
  • A list of recommendations for improving the user experience of the task you studied
  • A reflection on potential next steps for researching this interface (such as open questions you still have and how you might go about addressing them)

Late policy

  • Assignments submitted up to 24 hours late receive a deduction of one letter grade (10 percentage points).
  • Assignments submitted more than 24 hours late receive no credit.
  • The late penalty will be waived for up to 3 homework assignments.
    • However, this cannot be stacked (you can’t use more than one late day per assignment).
    • Even with this waiver, an assignment submitted more than 24 hours late receives no credit.