Evaluating a domain-specialist-oriented knowledge management system

We discuss the evaluation of software tools whose purpose is to assist humans to perform complex creative tasks. We call these creative task assistants (CTAs) and use as a case study CODE4, a CTA designed to allow domain specialists to manage their own knowledge base. We present an integrated process involving evaluation of usability, attractiveness and feature contribution, the latter two being the focus. To illustrate attractiveness evaluation, we assess whether CODE4 has met its objective of having users not trained in logical formalisms choose the tool to represent and manipulate knowledge in a computer. By studying use of the tool by its intended users, we conclude that it has met this objective. To illustrate feature contribution evaluation, we assess what aspects of CODE4 have in fact led to its success. To do this, we study what tasks are performed by users, and what features of both knowledge representation and user interface are exercised. We find that features for manipulating the inheritance hierarchy and naming concepts are considered the most valuable. Our overall conclusion is that those developing or researching CTAs would benefit from using the three types of evaluation in order to make effective decisions about the evolution of their products. (C) 2000 Academic Press.