Lessons from past, current issues, and future research directions in extracting the knowledge embedded in artificial neural networks

Active research into processes and techniques for extracting the knowledge embedded within trained artificial neural networks has continued unabated for almost ten years. Given the considerable effort invested to date, what progress has been made? What lessons have been learned? What direction should the field take from here? This paper seeks to answer these questions. The focus is primarily on techniques for extracting rule-based explanations from feed-forward ANNs since, to date, the preponderance of the effort has been expended in this arena. However the paper also briefly reviews the broadening overall agenda for ANN knowledge-elicitation. Finally the paper identifies some of the key research questions including the search for criteria for deciding in which problem domains these techniques are likely to out-perform techniques such as Inductive Decision Trees.