Evaluating an intelligent graphical user interface by comparison with human experts

This paper describes an intelligent graphical user interface (GUI) and its usability evaluation where the reasoning of the interface was compared to the reasoning of human experts. The GUI is called Intelligent File Manipulator (IFM) and it helps users with the management of their files and folders. IFM works in a similar way as a standard explorer such as Microsoft Windows 98 explorer, but it also incorporates intelligence. IFM is meant to act like a human expert that watches a user over the shoulder and offers advice spontaneously. It constantly monitors users' actions silently and generates automatic assistance when it diagnoses problematic situations. The generation of IFM's hypotheses about possible users' errors is based on an adaptation of a cognitive theory called Human Plausible Reasoning. As soon as the possible intended actions have been generated, the system uses a combination of the Human Plausible Reasoning theory with a multi-criteria decision making theory, called Simple Additive Weighting, in order to find the most appropriate action to be suggested to the user. The main focus of the evaluation of IFM is on the plausibility and usefulness of the advice selected through these methods as compared to the advice that human experts could provide if they watched the users working. The overall evaluation revealed that similarly to human experts, IFM could generate plausible hypotheses about users' errors to a satisfactory extent. (C) 2003 Elsevier B.V. All rights reserved.