The statistical power of replications in difference tests

The binomial test with N = nk observations is frequently used when n assessors each perform k replicated difference tests, for instance triangle tests. The power of this test, until now completely unknown for k > 1, is studied. Beta-binomial, generalized linear mixed and binomial mixture models are compared and shown to give similar results. This goes as well for model fits to real data as for power calculations. Corrected versions of the beta-binomial and the generalized linear mixed model are suggested rather than the standard versions. It is shown how the statistical power of the binomial test can easily be computed for the various approaches using Monte Carlo methods and standard software. An extreme version of the binomial mixture model can be seen as the common extreme case for all three approaches. This common extreme case scenario corresponds to the situation where each individual is assumed to be either a discriminator (having probability one of correct answer) or a non-discriminator (having probability c of correct answer). Although this is not the proper description of the data generating process it does provide a limit of power for a given combination of it and k. And importantly, this limit power calculation does not require any complicated model fitting. Tables of this limit power is provided for the triangle and duo-trio tests. A general result is that the loss of power compared to the independent case is remarkably small. In other words, given the number of assessors, the power of the binomial test can be increased considerably by just a few replications. (C) 2003 Elsevier Science Ltd. All rights reserved.