An experimental framework for interactive information retrieval and digital libraries evaluation

Evaluation of digital libraries assesses their effectiveness, quality and overall impact. In this paper we propose to use the Daffodil system as an experimental framework for the evaluation and research of interactive IR and digital libraries. The system already provides a rich set of working services and available information sources. These services and sources can be used as a foundation for further research going beyond basic functionalities. Besides the services and sources, the system supports a logging scheme for comparison of user behavior. In addition, the system can easily be extended regarding both services and sources. Daffodil's highly flexible and extensible agent-based architecture allows for easy integration of additional components, and access to all existing services. Finally, the system provides a user-friendly graphical interface and facilitating services for log generation and analysis. The experimental framework can serve as a joint theoretical and practical platform for the evaluation of DLs, with the long-term goal of creating a community centered on interactive IR and DL evaluation.