Ethics in evaluation with people

Intuition

Almost every HCI evaluation uses people, and people can be embarrassed, stressed or exposed. The checklist exists so the study does not cost the participant more than they agreed to.

Ethics approval comes first because evaluation involves humans

The lecture says evaluation in HCI almost always involves humans. You inform participants of their rights, protect them from physical or emotional endangerment, and protect their privacy, including how data is collected and treated. Ethics approval must be obtained before the study. Check your own organisation’s rules.

The University Human Ethics Policy gives four reasons: protect the welfare, rights, dignity and safety of participants; protect researchers’ rights to conduct legitimate investigation; protect the University’s reputation; and minimise claims of negligence. Human research is research conducted with or about people, their biological materials or information. It covers surveys, interviews and focus groups; psychological, physiological or medical testing or treatment; being observed; accessing personal documents or other materials; collecting biological materials; and accessing personal information from a published or unpublished source. The lecture also points to the NHMRC National Statement on Ethical Conduct in Human Research.

Three phases, each with comfort, privacy and time

Exam detail

Before: do not waste time (pilot studies, be prepared). Make users comfortable: only the system is tested, not the user; software may have problems; they can stop at any time. Maintain privacy: results anonymised where applicable. Inform: explain what is recorded (video, audio, data logging), answer questions but avoid bias. Do not coerce: obtain informed consent.

During: no unnecessary tasks. Early success (pre-trials), relaxed atmosphere, sufficient breaks, tasks handed out one at a time, no displeasure shown, avoid disruptions, stop if the participant shows discomfort. Privacy: no external people present.

After: thank the user and say they helped. Answer remaining questions (for example, something that could have biased them). Report without compromising privacy, share audio or visual data only with expressed permission, store data securely (the University’s Research Computing Optimised Storage, RCOS).

Pitfall

Answering a participant’s questions is allowed, but “avoid bias” is in the same line. Hold back anything that would change how they perform, and explain it afterwards.

Recall

Which three themes recur in before, during and after?

Do not waste the user’s time, make users feel comfortable, and maintain privacy (plus informing the user and not coercing before the session).