Multi-criteria methodology contribution to the software quality evaluation

Industrial evaluations of COTS software largely used the quality models provided by the international standards. But the context and objectives of COTS evaluations are fundamentally different than those primarily defined by the standards. Several key issues are often forgotten: (1) the existence of several evaluators and several quality models sharing common factors, criteria and measures, (2) the purpose of the evaluation model, (3) measures of different types, and (4) the recursive nature of the model since each node is an evaluation model itself. We had the occasion to study the results of real standard-based COTS evaluations. Faced with the difficulties to exploit them, we experimented the use of multi-criteria methodology. This work allows us to understand some of the problems generated by the application of the standards to COTS evaluations, and to propose new principles for evaluating software quality that should be considered in an evolution of the standards. This paper reports our experiment.