Measuring card sort orthogonality

Card sorts can be used to study the way human subjects acquire and organize conceptual knowledge. Analyses of card sorts often involve subjective examination of criteria or category names or using clustering techniques, neither of which lend themselves well to direct statistical analysis. This paper defines NMST, a quantitative measure of knowledge discrimination based on repeated, single-criterion card sorts that is independent of criteria or category names and that is particularly amenable to statistical analysis. The NMST measure is illustrated by applying it to a particular data set collected in a large multinational card sort study of subjects, where the knowledge area comes from a first-year programming course. Applied to this data set, the NMST measure is shown to distinguish, with statistical significance, between a set of subjects with only an introduction to programming and a set of subjects who have completed the equivalent of a bachelor's degree or higher in a computing-related discipline.