MADCAP: a graphical method for assessing risk scoring systems

Objective: We set out to develop a method for assessing the performance of clinical risk models over the spectrum of risks and to assess the performance of the EuroSCORE risk model used in cardiac surgery. Methods: We developed a graphical method for assessing the performance of clinical risk models over the spectrum of risks. To illustrate the technique, we analysed retrospective data concerning 9268 patients that underwent cardiac surgery and for whom both the additive EuroSCORE prediction of risk of morality and vital status at 30 days were available. Results: The graphical tool developed, called MADCAP (Mean Adjusted Deaths Compared Against Predictions), can be used to highlight systematic features of the performance of a clinical risk model. Its use in the current study indicates that the additive version of the EuroSCORE model seems to underestimate risk amongst tow-risk cases (0% and 1%). Otherwise the score systematically favours risk avoiding behaviour as the risk model underestimates mortality for 2-6% prediction but not at 7% and above. Conclusion: The robustness of case-mix adjusted audit is dependent on the performance of the risk scoring system over the entire spectrum of risk. If we are to use risk adjustment of mortality rates when comparing outcomes obtained by different units or individual surgeons, it is essential that we continually review the performance of the risk adjustment method. The MADCAP method presented here provides a useful toot to this end. (c) 2006 Elsevier B.V. All rights reserved.