Measurement of the reliability of pharmacy staff and simulated patient reports of non-prescription medicine requests in community pharmacies

Background: The use of simulated patients (SPs) in pharmacy practice research has become an established method to observe practice. The reliability of data reported using this method in comparison to pharmacy staff selfreported behaviour has yet to be ascertained. Objective: To compare the inter-rater agreement of pharmacy staff and SP-reported data to researcher-reported data from audio recordings of SP encounters. Methods: A dataset of 352 audio-recorded SP encounters was generated in March-October 2015 by 61 undergraduate pharmacy students completing SP visits to 36 community pharmacies in Sydney, Australia. Post-visit scores were recorded on data collection forms by SPs. Staff completed self-assessments on identical forms immediately after the encounter. Two-hundred-and-seventy visits were randomly selected as the sample for this study, where the researcher independently scored encounters via audio recordings. Inter-rater agreement was calculated through intra-class correlation (ICC) and weighted kappa analyses. Results: Analysis of staff scores returned ICC values of 0.48 (95% CI:0.38-0.56; p < 0.001) for information gathering and 0.63 (95% CI:0.55-0.70; p < 0.001) for total score. Weighted kappa for information rating was 0.30 (95% CI:0.21-0.38; p < 0.001) and 0.43 (95% CI:0.34-0.51; p < 0.001) for overall outcome. ICC values for SPs were 0.91 (95% CI:0.88-0.93; p < 0.001) and 0.90 (95% CI:0.87-0.92; p < 0.001) for information gathering and total scores respectively. Weighted kappa values were 0.44 (95% CI:0.37-0.52; p < 0.001) for information rating and 0.63 (95% CI:0.55-0.70; p < 0.001) for overall outcome. Conclusion: Pharmacy staff self-reported their behaviour with a poor degree of reliability. Conversely, SPs had a high level of agreement with the researcher scoring from audio recordings. Disagreement for both groups of raters was most apparent in rating the information provided and overall appropriateness of outcome. Future research should investigate this discrepancy between staff-reported behaviour and actual behaviour and consider the implications of this discrepancy in the interpretation of self-reported data.