The Issue of Reliability in Software Mediated Process Assessments

This paper reports on a reliability analysis of a Software-mediated Process Assessment (SMPA) decision support tool as it was used at three sites. The tool was used at two sites as part of its development process. Following its evaluation, further development included the formulation of assessment questions and knowledge items to complete the requirements for all ISO/IEC 20000 processes. The SMPA was deployed at a third site, a large financial services firm. These three case studies brought up the issue of the reliability of the capability ratings and the value of the reliability measure (coefficient of variation) incorporated in the SMPA. Further data collection when undertaking assessments is required to have better assurance of the results given the variability of subjects. In addition the future of ISO/IEC 15504/ISO/IEC 330xx and ISO/IEC 20000 and how the SMPA will need to change to maintain alignment is raised.