Cognitive Evaluation of Machine Learning Agents

Advances in applying statistical Machine Learning (ML) led to several claims of human-level or near-human performance in tasks such as image classification & speech recognition. Such claims are unscientific primarily for two reasons, (1) They incorrectly enforce the notion that task-specific performance can be treated as manifestation of General Intelligence and (2) They are not verifiable as currently there is no set benchmark for measuring human-like cognition in a machine learning agent. Moreover, ML agent's performance is influenced by knowledge ingested in it by its human designers. Therefore, agent's performance may not necessarily reflect its true cognition. In this paper, we propose a framework that draws parallels from human cognition to measure machine's cognition. Human cognitive learning is quite well studied in developmental psychology with frameworks and metrics in place to measure actual learning. To either believe or refute the claims of human-level performance of machine learning agent, we need scientific methodology to measure its cognition. Our framework formalizes incremental implementation of human-like cognitive processes in ML agents with an implicit goal to measure it. The framework offers guiding principles for measuring, (1) Task-specific machine cognition and (2) General machine cognition that spans across tasks. The framework also provides guidelines for building domain-specific task taxonomies to cognitively profile tasks. We demonstrate application of the framework with a case study where two ML agents that perform Vision and NLP tasks are cognitively evaluated. (C) 2020 Elsevier B.V. All rights reserved.