DELPHI, THE ISSUE OF RELIABILITY - A QUALITATIVE DELPHI STUDY IN PRIMARY HEALTH-CARE IN THE NETHERLANDS

Problems arise when reliability and accuracy of the Delphi technique are assessed. So far, reliability has been evaluated by comparing two or more Delphi studies on the same subject, using Pearson, kappa, or rank-order coefficients. However, this strategy is not very effective or informative. When numerical ratings of repondents are available and these ratings are normally distributed, reliability of Delphi can be evaluated in a more accurate and effective way by means of the intraclass correlation coefficient. If the investigator expects any systematic effects of factors characterizing the design of a study, these effects can be extracted from the error variance and analyzed separately. A second advantage of the intraclass correlation coefficient is that it can be derived from the ratings of only one group of respondents. Collecting extra data is not necessary. By using this strategy, the evaluation of reliability becomes feasible in individual applications. The concept is systematically elaborated in a Delphi study carried out to develop evaluation criteria for the performance of family physicians consulted by patients having nonspecific abdominal complaints.