Planning an evaluation: why, what, where, when

Evaluate to judge features, effects, surprises and competitors

The lecture gives four reasons. Judge system features and functionality: does it facilitate users’ tasks, match requirements and offer the right features? Judge effects on users: how easy it is to learn and use, how users feel. Discover unforeseen problems: unexpected or confusing situations. Compare your solution against competitors, which matters to marketing and sales.

You can evaluate the conceptual model, early and subsequent prototypes of a new system, and the final product. Evaluating an early design uses lo-fi prototypes to find and fix issues in advance. Evaluating usability applies to a product that is close to final and feature rich and gives comparative results (for example, X is a percent faster than Y). Evaluating for user groups checks how a product works for new markets, such as an Australian product moving to the African continent.

Lab and field trade control for realism

Where you evaluate: usability lab, research lab, naturalistic settings or remote study. A usability lab is a testing room built for usability testing, instrumented with cameras, microphones and data recording, with a separate observation room joined by a one-way mirror.

Evaluation runs through the lifecycle and follows seven steps

When to evaluate: during iterative design (does the design match requirements, what are the problems), before deployment for acceptance testing (does the system meet expected performance), and continuously after deployment, called “permanent beta” (in the wild, bug reports, field studies).

Use a combination of methods. Controlled methods test hypotheses about specific features. Uncontrolled methods give insight into experience in daily life. A cognitive walkthrough can test-run a prototype before the actual lab usability test.