Analysing data: quantitative and qualitative

The type of data decides the type of analysis

Intuition

Time on task gives you a number to average. A comment such as “the class was interesting” gives you a sentence to interpret. Each needs different tools and supports a different kind of claim.

Compare

Quantitative

Numbers, such as years of experience or time taken for a task. Analysed with numerical methods to find size, magnitude or amount, for example statistical analysis.

Qualitative

Words (quotes, descriptions) and images (vignettes, photos), such as “the class was interesting”. Analysis expresses the nature of elements and is represented as themes, patterns and stories.

Question design shapes what you can conclude. Open questions are analysed answer by answer, closed questions quantitatively, and fixed alternative answers restrict what can be said in the findings.

Quantitative analysis cleans, codes, describes and tests

Mechanism

1. Clean: screen for errors (an age of 223), run reasonableness checks (age 24 with technical experience 30), group data for the same participant across channels, and fix formatting (9 versus nine).
2. Code: convert the original data to numeric form.
3. Describe: mean (add and divide by the count), median (middle value when ranked), mode (most frequent figure), standard deviation (amount of variation).
4. Test: a significance test for comparing means, and correlation testing such as Pearson’s product moment correlation to identify relationships between factors, for example age, computing experience and target selection speed.

Tools: Excel (simple, quick), SPSS (full-featured with a GUI, expensive) and R (full-featured, scriptable, free, similar to Python or MATLAB, but you must learn to program in it). The lecture slides on tests and the coding table were figures, so the specific tests beyond the two named are not recorded.

Qualitative analysis looks for incidents, themes and categories

• Critical incidents focus analysis on significant events. In an evaluation study they are signalled by a comment, silence or a look of puzzlement, and the follow-up depends on the severity of the problem.
• Identifying themes is inductive: themes emerge from the data and from the observation framework if one was used. It suits exploratory studies, and an affinity diagram is the tool.
• Categorising data is deductive: a top-down approach where categories are set beforehand and the data is classified into them.

In practice, analysis combines inductive and deductive approaches.

An affinity diagram is built bottom-up. Write each data point on a Post-it, convert all data, let team members define their own groups, affix notes to a wall as groups emerge (discarding duplicates or near-duplicates), and record the groups and headings.

Pitfall

Inductive means bottom-up (themes emerge). Deductive means top-down (categories first). The affinity diagram is inductive. A pre-set coding scheme is deductive.

Analytical frameworks help interpret qualitative data. Interaction analysis studies interactions between people and between people and artefacts, based on empirical observations such as video, as an inductive team process: content is logged, material is extracted, classified or removed, instances of a salient event are assembled and played one after another, and the team studies them together. It suits evaluating a prototype during a usability test. Content analysis suits identifying user preferences. Tools: nVivo (annotating and coding PDFs, photos, video and audio, with the latest version coding directly onto video and audio without transcription) and Atlas.ti.

Recall

Which analysis is inductive and which deductive, and what tool supports the inductive one?

Identifying themes is inductive and supported by an affinity diagram. Categorising data is deductive with categories fixed beforehand.