08-03 · Week 8
Experiments, variables and experimental designs
How an experiment differs from a usability test, independent and dependent variables, between, within and matched designs, and the seven steps.
- Contrast usability testing with experiments for research.
- Identify independent and dependent variables in a scenario.
- Choose between different-participants, same-participants and matched-participants designs and name the weakness of each.
- Order the seven steps of running an experiment.
15 min readJump to the quiz ↓
Intuition
An experiment changes one thing and measures another. The hard part with people is that they learn as they go, so who does what, and in which order, decides whether the result means anything.
Usability testing is applied experimentation, research experiments are stricter
Usability testing is applied experimentation: developers check that the system is usable by the intended population by collecting data about participants’ performance on prescribed tasks. Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more variables.
Compare
Usability testing
Experiments for research
Variables: the researcher moves one and measures the other
An experiment tests a hypothesis that predicts a relationship between two or more variables. The independent variable is manipulated by the researcher (the test conditions). The dependent variable is the change the manipulation causes (performance). Typical designs have one or two independent variables. The experimental design sets up conditions so other variables are held constant and cannot influence the findings.
The lecture’s example tested whether types of ambient noise affect smartphone interaction in a controlled lab study with 24 participants (12 female, 12 male), 40 minutes each. Noise types were music (fast and slow tempo), urban noise (indoor, outdoor), speech (meaningful English, meaningless Kazakh) and silence as the control, all at 60 dB. Independent variables: noise type, gender, age. Dependent variables: for target acquisition, task completion time and touch offset size; for visual search, search time, memorisation time and error rate; for text entry, time per character and error rate. It used a within-subjects design with counter-balanced conditions and randomised task order to avoid sequence effects.
Three designs trade participants against ordering effects
Humans learn from experience, so you must decide who takes part in which condition. Experience in one condition affects performance in another.
Exam detail
Different-participants (between-subjects): one group is randomly allocated to the conditions, so different participants perform in different conditions. Needs many participants to minimise individual differences in experience and expertise. Pre-test to find participants who differ strongly.
Same-participants (within-subjects): all participants perform in all conditions. Must counter-balance to reduce ordering effects, where learning from a previous task affects the next.
Matched-participant (pair-wise): participants matched in pairs by characteristics such as expertise and gender, each pair randomly allocated to conditions. Does not consider other important variables that may influence results.
Seven steps: (1) determine goals, explore the questions, formulate a hypothesis; (2) design the experiment, define variables; (3) choose subjects; (4) run a pilot; (5) iteratively improve the design; (6) run the experiment; (7) interpret results to accept or reject the hypothesis.
Try it: worked example
A within-subjects study has two conditions, A and B, and four participants. How can the order be counter-balanced?
Show answer and steps
AnswerHalf do A then B, half do B then A, assigned randomly.
- In a within-subjects design every participant does both conditions, so whichever comes second benefits from practice.
- Split the four participants into two groups of two.
- Group 1 does A then B. Group 2 does B then A.
- Any practice effect now applies equally to both conditions, so it does not systematically favour one.
Pitfall
Between-subjects means different participants in different conditions. Within-subjects means the same participants in all conditions. Questions often describe a scenario without naming the design; look at who does what.
Recall
Which design needs counter-balancing, and which needs many participants?
Within-subjects needs counter-balancing; between-subjects needs many participants.
Check yourself
Source
Lecture 8 PDF