Evaluation of verification tools for knowledge-based systems

Validation has emerged as a significant problem in the development of knowledge-based systems (KBS). Verification of KBS correctness and completeness has been cited as one of the most difficult aspects of validation. A number of software tools have been developed to perform such verification, but none of these are in widespread use. One of the reasons for this is that little quantitative evidence exists to demonstrate the effectiveness of the tools. This paper presents an experimental study of three KBS verification tools: a consistency checker, a completeness checker and a testing tool(for correctness). The tools are evaluated on their ability to reveal plausible faults seeded into a complex, realistic KBS application. The cost of using the tools is also measured. It is shown that each tool is independently effective at detecting certain kinds of fault and that the capabilities of the tools are complementary-a result not revealed by previous studies. (C) 1997 Academic Press Limited.