Analytics, A/B testing and model-based evaluation

Intuition

Sometimes the cheapest way to watch users is to let the software do it, or to skip the users and calculate. These three methods do that.

Analytics records what people do without anyone watching

A variety of user actions can be recorded automatically: key presses, time spent searching a web page, looking at help systems. It is unobtrusive provided system performance is not affected. Large volumes are logged automatically and then explored and analysed. The ethical issue is observing without knowledge.

Web analytics is a form of interaction logging that analyses user activity on a website: total number of people, length of stay and content visited. The outcomes improve designs, because when designs do not meet users’ needs, users do not return (one-time users). Learning analytics applies web analytics to education, for example learner activity in MOOCs and Open Education Resources. Visual analytics displays and manipulates big data visually.

A/B testing compares two versions on two groups

A/B testing is a large-scale experiment and another way to evaluate a website, application or mobile app. It is often used for design changes on social media applications. It compares how two groups of users perform on two versions of a design.

Model-based evaluation predicts performance without users

Predictive models evaluate a system without users present, using formulas to derive measures of user performance. The lecture’s example is Fitts’ Law: the time taken to hit a screen target depends on the distance of the cursor and the size of the target. Its cited research compares dogs’ ability to use a touchscreen with human performance.

Optional aside

The lecture states Fitts’ Law in words only. In the textbook it is written as a formula with logarithmic distance over width; the slides do not give one, so you are not expected to recall it from this lecture.

Recall

Which of the three methods needs no users at all?

Model-based evaluation. Analytics and A/B testing both use real users, although analytics can run without them knowing, which is its ethical problem.