What matched comparisons can and cannot tell us: The case of cochlear implants

Objectives: To examine the conclusions and possible misinterpretations that may or may not be drawn from the "outcome-matching method," a study design recently used in the cochlear implant literature. In this method, subject groups are matched not only on potentially confounding variables but also on an outcome measure that is closely related to the outcome measure under analysis. For example, subjects may be matched according to their speech perception scores in quiet, and their speech perception in noise is compared. Design: The present study includes two components, a simulation study and a questionnaire. In the simulation study, the outcome-matching method was applied to pseudo-randomly generated data. Simulated speech perception scores in quiet and in noise were generated for two comparison groups, in two imaginary worlds. In both worlds, comparison group A performed only slightly worse in noise than in quiet, whereas comparison group B performed significantly worse in noise than in quiet. In Imaginary World 1, comparison group A had better speech perception scores than comparison group B. In Imaginary World 2, comparison group B had better speech perception scores than comparison group A. The outcome-matching method was applied to these data twice in each imaginary world: 1) matching scores in quiet and comparing in noise, and 2) matching scores in noise and comparing in quiet. This procedure was repeated 10,000 times. The second part of the study was conducted to address the level of misinterpretation that could arise from the outcome-matching method. A questionnaire was administered to 54 students in a senior level course on speech and hearing to assess their opinions about speech perception with two different models of cochlear implant devices. The students were instructed to fill out the questionnaire before and after reading a paper that used the outcome-matching method to examine speech perception in noise and in quiet with those two cochlear implant devices. Results. When pseudorandom scores were matched in quiet, comparison group A's scores in noise were significantly better than comparison group B's scores. Results were different when scores were matched in noise: in this case, comparison group B's scores in quiet were significantly better than com parison group Xs scores. Thus, the choice of outcome measure used for matching determined the result of the comparison. Additionally, results of the comparisons were identical regardless of whether they were conducted using data from Imaginary World I (where comparison group A is better) or from Imaginary World 2 (where comparison group B is better). After reading the paper that used the outcome-matching method, students' opinions about the two cochlear implants underwent a significant change even though, according to the simulation study, this opinion change was not warranted by the data. Conclusions: The outcome-matching method can provide important information about differences within a comparison group, but it cannot be used to determine whether a given device or clinical intervention is better than another one. Care must be used when interpreting the results of a study using the outcome-matching method.