Psychometric properties of a simple measure of conceptual knowledge: The concept retrieval technique

The objective of the present study was to establish reliability and validity evidence of a simple procedure to measuring conceptual knowledge, the Concept Retrieval Technique (CRT). The CRT procedure requires test takers to freely retrieve concepts from memory they feel are relevant to a given topic, with raters awarding one mark for each correctly retrieved concept when matched against a target word list. Four studies were conducted. In Study 1 (N = 73), inter-rater agreement for the marking procedure of the CRT was determined by means of Cohen’s kappa. The kappa was κ = .85. Study 2 (N = 164) explored how consistent inter-rater agreement was across different subjects, age groups, and raters. Mean kappa was κ = .92 suggesting that the CRT can reliably be marked. Study 3 was concerned with establishing the convergent and discriminant validity of the CRT (N = 55). The correlation between the CRT scores and scores on essay-type items was r = .69 and between assessments that measured other ability domains r = .04 to .21 (n.s.). In Study 4, an experiment was conducted to determine the construct validity of the CRT (N = 45). Participants either acquired, or did not acquire, new knowledge of a particular topic and this manipulation was consistently reflected in their CRT scores. The results of the studies suggest that the CRT is a reliable and valid procedure to measure conceptual knowledge. Implications for education are discussed.