Subjective evaluation of decision support systems using multiattribute decision making (MADM)

Structured evaluation of decision support system (DSS) development stages may be found in development plans, however, formal and systematic evaluations of "real world" DSS are rarely performed throughout the creation process. As a result, we relinquish the chance to gain valuable information about what potential customers think about the system, how well the code is written, the extent to which the system actually supports decision making, etc. Andriole (1982) estimated that reliable evaluations were conducted on about only 10% of government-sponsored DSS development projects. Multiattribute decision making (MADM) is presented herein as a mechanism to account for different attributes or system effectiveness measures (SEMs) considered important when evaluating a DSS and as a means for combining the SEMs into a comprehensive measure of value, that is, a single overall assessment of system utility. MADM is an appropriate tool for evaluating DSS in that it is capable of distinguishing between satisfactory and unsatisfactory systems, or components of a system. A case study is presented showing how MADM was used to evaluate two potential Graphical User Interfaces (GUIs) for the USDA-ARS Great Plains Framework for Agricultural Resource Management (GPFARM) DSS.