OpenSearch: Lessons Learned from an Online Evaluation Campaign

被引：11

作者：

Jagerman, Rolf ^{[1
]}

Balog, Krisztian ^{[2
]}

De Rijke, Maarten ^{[1
]}

机构：

[1] Univ Amsterdam, Sci Pk 904, NL-1098 XH Amsterdam, Netherlands

[2] Univ Stavanger, Kristine Bonnevies Vei 22, N-4021 Stavanger, Norway

来源：

ACM JOURNAL OF DATA AND INFORMATION QUALITY | 2018年 / 10卷 / 03期

关键词：

Living labs; online evaluation;

D O I：

10.1145/3239575

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

We report on our experience with TREC OpenSearch, an online evaluation campaign that enabled researchers to evaluate their experimental retrieval methods using real users of a live website. Specifically, we focus on the task of ad hoc document retrieval within the academic search domain, and work with two search engines, CiteSeerX and SSOAR, that provide us with traffic. We describe our experimental platform, which is based on the living labs methodology, and report on the experimental results obtained. We also share our experiences, challenges, and the lessons learned from running this track in 2016 and 2017.

引用

页数：15

共 50 条

[41] Lessons learned from an evaluation of referrals to the emergency department
Roee Gorodetzer
Evan Avraham Alpert
Zvika Orr
Shifra Unger
Todd Zalut
Israel Journal of Health Policy Research, 9
[42] LESSONS LEARNED ABOUT PUBLIC HEALTH FROM ONLINE CROWD SURVEILLANCE
Hill, Shawndra
Merchant, Raina
Ungar, Lyle
BIG DATA, 2013, 1 (03) : 160 - 167
[43] An Alternate Reality for Education? Lessons to be Learned from Online Immersive Games
Moseley, Alex
INTERNATIONAL JOURNAL OF GAME-BASED LEARNING, 2012, 2 (03) : 32 - 50
[44] Crisis Planning for Online Students: Lessons Learned from a Major Disruption
Holzweiss, Peggy C.
Walker, Daniel W.
Chisum, Ruth
Sosebee, Thomas
ONLINE LEARNING, 2020, 24 (02): : 22 - 37
[45] Understanding and supporting online communities of practice: lessons learned from Wikipedia
Xiaoli Zhao
M. J. Bishop
Educational Technology Research and Development, 2011, 59 : 711 - 735
[46] ONLINE COMPUTER PHARMACOKINETICS PROGRAM - LESSONS LEARNED FROM ITS FAILURE
BAER, DM
KOTSCHI, ML
CLINICAL CHEMISTRY, 1995, 41 (04) : 491 - 494
[47] Lessons Learned From an Online Study with Dual-smoker Couples
Choi, Seung Hee
Mitchell, Jason
Lipkus, Isaac
AMERICAN JOURNAL OF HEALTH BEHAVIOR, 2017, 41 (01): : 61 - 66
[48] Understanding and supporting online communities of practice: lessons learned from Wikipedia
Zhao, Xiaoli
Bishop, M. J.
ETR&D-EDUCATIONAL TECHNOLOGY RESEARCH AND DEVELOPMENT, 2011, 59 (05): : 711 - 735
[49] Moving conferences online: lessons learned from an international virtual meeting
Stefanoudis, Paris, V
Biancani, Leann M.
Cambronero-Solano, Sergio
Clark, Malcolm R.
Copley, Jonathan T.
Easton, Erin
Elmer, Franziska
Haddock, Steven H. D.
Herrera, Santiago
Iglesias, Ilysa S.
Quattrini, Andrea M.
Sigwart, Julia
Yesson, Chris
Glover, Adrian G.
PROCEEDINGS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES, 2021, 288 (1961)
[50] LESSONS LEARNED FROM ONLINE INTERVENTIONS ON ANXIETY AND DEPRESSION FOR CHILDREN AND ADOLESCENTS
vanWingerden, E.
vanOosterom-Calo, R.
Kempelmann, G.
Sterkenburg, P.
JOURNAL OF INTELLECTUAL DISABILITY RESEARCH, 2021, 65 (08) : 758 - 758

← 1 2 3 4 5 →