Evaluation of Context-Aware Recommendation Systems for Information Re-Finding

In this article we evaluate context-aware recommendation systems for information re-finding by knowledge workers. We identify 4 criteria that are relevant for evaluating the quality of knowledge worker support: context relevance, document relevance, prediction of user action, and diversity of the suggestions. We compare 3 different context-aware recommendation methods for information re-finding in a writing support task. The first method uses contextual prefiltering and content-based recommendation (CBR), the second uses the just-intime information retrieval paradigm (JITIR), and the third is a novel network-based recommendation system where context is part of the recommendation model (CIA). We found that each method has its own strengths: CBR is strong at context relevance, JITIR captures document relevance well, and CIA achieves the best result at predicting user action. Weaknesses include that CBR depends on a manual source to determine the context and in JITIR the context query can fail when the textual content is not sufficient. We conclude that to truly support a knowledge worker, all 4 evaluation criteria are important. In light of that conclusion, we argue that the network-based approach the CIA offers has the highest robustness and flexibility for context-aware information recommendation.