Estimating Accuracy and Confidence Interval of an Intelligent Diagnostic Reasoner System

Intelligent Diagnostic Reasoning System (IDRS), developed by Lockheed Martin Simulation, Training & Support (LM STS), implements a Bayesian model that is able to reduce the time and cost to diagnose failures by isolating faults[1]. As is the case with all learning systems, the quality of diagnosis is expected to increase with time as more data is presented and more knowledge is absorbed by the system. Since learning is an ongoing process, at any given time, we would like to get an estimate on the accuracy of the system given the data it has seen so far and the Bayesian Network structure it started with. In this paper we describe one approach for estimating the accuracy of diagnosis in an IDRS system. We also outline a method to compute the confidence interval on the estimated accuracy of the system. In addition, we present a way to define confidence intervals for individual probabilities of diagnosing faults. These measures combined allow us to appropriately quantify confidence in a learning system. Finally, we illustrate results from our simulation on accuracy estimation and determination of confidence intervals in IDRS using field data.