Usability testing: roles, tasks and participants

Intuition

A usability test is a small stage play with a script. Everyone has a part, the participant gets a task, and the test only tells you something if every participant plays under the same conditions.

A usability test records typical users doing typical tasks

Usability testing involves recording the performance of typical users doing typical tasks. Users are observed and timed, data is recorded on video and key presses are logged, and satisfaction is evaluated with questionnaires and interviews. The lecture organises the practical side as five questions: what should my team do, what should the participant do, who do I recruit, how many do I need, how do I prepare (including paperwork and informed consent, covered next).

Five roles share the work

All members should be encouraged to take part. The roles:

Exam detail

• Facilitator: in the lab with the participant; plans and executes the session, sets up the lab, puts the participant at ease; must have people skills.
• Prototype executor: moves a lo-fi prototype through its paces as users interact; needs thorough technical knowledge; poker face, says nothing during the session.
• Quantitative data collector: time to complete a task; number and type of errors per task; errors per unit of time; navigations to online help or manuals. Record into a spreadsheet for lo-fi (stopwatch and counter), programmed for hi-fi.
• Qualitative data collector: observation notes, critical incidents, think-aloud comments.
• Supporting actors: if the setting needs the participant to interact with someone (calling a client on the phone), and managing props other than the prototype.

The Week 9 worksheet runs this as a think-aloud session: one facilitator, one researcher writing down comments and observations, one prototype executor, and a user from another team. Tasks tell users what to do without giving instructions on how.

Tasks say what to do, never how

Usability tests are usually task-based. Use representative, frequent and critical tasks for the key work role and user class each participant represents. Prepare task descriptions and UX target metrics to guide data collection and to compare observed results. Test conditions are the same for every participant. A task description says what to do with no hints about how (the lecture’s video example: “Show me how you eat this”).

Participants are representative, and the number is a practical choice

Find representative users, usually outside your team and project organisation. Recruitment sources and screening: people around you, ads in public spaces, announcements at user group or professional organisation meetings that match your user class, and temporary employment agencies.

The number is a practical issue shaped by schedule, availability and cost. The famous rule of thumb is three to five participants, from the claim that five evaluators find about 80 percent of UX problems. It does not always apply. Typically 5 to 10 take part, or you test until no new insights appear.

Pitfall

Do not mix the two “five” statistics. The heuristic evaluation figure is five expert evaluators finding 75 to 80 percent (Nielsen and Mack, 1994). The usability testing figure is a three to five participant rule of thumb with about 80 percent. Both are lecture content.

Recall

Which role keeps silent, which role makes the participant feel at ease?

The prototype executor keeps silent (poker face); the facilitator puts the participant at ease.