Value of search results as a whole as a measure of information retrieval performance

Value of search results as a whole, a utility measure, was found to be the best single measure or indicator of interactive information performance (success) among the 20 measures selected for study (Su, 1991, p. 109; 1992, p. 514). Value of search results as a whole is a measure which asks for the user's rating on the value or usefulness of search results taken as a whole for meeting the user's need or resolving the user's problem, based on a Likert 7-point scale. The study suggests that the single measure, value of search results as a whole, provides an easy and simple way for system comparison and choice, and it eliminates the problems of IR evaluation with multiple measures. However, value of search results as a whole is a measure which provides a numeric basis for comparing system performance but no information on why one set of search results is rated more useful than others or how search results or systems can be improved to be more useful or successful. To further our understanding of the measure, value of search results as a whole, and enhance its usefulness as a measure of information retrieval performance, the current paper aims to examine two issues: (1) What are the conceptual categories or dimensions of the users' reasons for assigning particular ratings on the value of search results? and (2) What are the relationships between these dimensions of value and the dimensions of success identified in the earlier study (Su, 1991)? The earlier study (Su, 1991, pp. 82-90) was conducted to investigate the appropriateness of 20 measures for evaluating interactive information retrieval performance (success), representing four major evaluation criteria. The user's judgment of overall system success was used as the devised criterion measure with which all other 20 measures were to be correlated (p. 64). A sample of 40 end-users with individual information problems from an academic environment were observed, interacting with six professional intermediaries searching on their behalf in large operational systems at the users' own costs. A search was conducted for each individual problem in the user's presence and with the user's participation as it was normally done in the particular environment. Quantitative data consisting of scores for all measures studied and verbal data containing users reasons for assigning certain ratings to selected measures were collected. The portion of the verbal data including users' reasons for assigning particular value ratings from the previous study will be fully transcribed and content analyzed for the current study. Preliminary findings pertaining to the two research questions will be presented and implications will be addressed.