A TASK-ORIENTED APPROACH TO RETRIEVAL-SYSTEM EVALUATION

There is a need for improved methods to evaluate the effectiveness of end-user information retrieval systems. We performed a task-oriented assessment of two MEDLINE searching systems, one which promotes Boolean searching on human-indexed thesaurus terms and the other natural language searching on words in the title, abstract, and indexing terms. Each was used by medical students to answer clinical questions. Students were able to use each system successfully, with no significant differences in questions correctly answered, time taken, relevant articles retrieved, or user satisfaction between the systems. This approach to evaluation was successful in measuring effectiveness of system use and demonstrates that both types of systems can be used equally well. with minimal training.