Card sort analysis: similarity and clustering

The matrix answers “do people think these two belong together?” one pair at a time. The dendrogram answers “what groups would most people make?” by building them up from the most agreed pairs. Read them together: a tidy dendrogram over a matrix of pale cells means people agreed on little and the clustering forced a shape anyway.

The procedure

1. Take each participant's groups. Cards left unsorted or put in the unsure pile don't count as a group.
2. For every pair of cards, count the participants who sorted both, and the participants who put them in the same group.
3. Similarity is the second count over the first. When nobody sorted both cards, the cell is blank rather than zero.
4. Turn similarity into distance (1 minus similarity), treating a blank as no similarity.
5. Cluster with average linkage (UPGMA): join the two closest clusters, recompute distances as the average over every pair of cards between clusters, and repeat until one cluster is left.
6. Break ties between equal distances on the lowest original card index, so the dendrogram is the same on every load.
7. Order the matrix by the dendrogram's leaves, so groups show as dark blocks along the diagonal.

Average linkage is the UPGMA method Sokal and Michener described in 1958 [1]. Our clustering gives the same merge heights as SciPy's average linkage [3] and R's hclust with method average, which is how it's tested. Open sorts also offer label standardisation (grouping participants' category names that mean the same thing) and the Actual Agreement Method, a tree built only from groups participants really made.

A worked example

The sample open sort on the surveys and IA page: 38 participants sorted 12 cards from the Fenwick Grocers app. The five pairs people agreed on most:

Every participant sorted every card here, so each similarity is simply the together count over 38. Clustering then joins the closest pairs first. The first three joins:

Agreement is 1 minus the height at which the join happens. Cutting the dendrogram at 50% agreement, as the example on the surveys and IA page does, keeps only groups that at least half the participants would broadly recognise.

How many participants

Tullis and Wood (2004) [2] found that similarity matrices from 20 to 30 participants correlated closely with those from much larger samples. For an open sort feeding a navigation decision, 30 is a sensible target, and the price doesn't change if you collect more.

Used on Card sorting. Stroop is the all-in-one research and testing platform. Run usability tests, prototypes, surveys, tree tests, diaries and IATs in one study, and get results back transcribed, scored and charted, with every finding linked to the session it came from.