OpenSearch: Lessons Learned from an Online Evaluation Campaign

被引：11
|
作者：
Jagerman, Rolf [1 ]
Balog, Krisztian [2 ]
De Rijke, Maarten [1 ]
机构：
[1] Univ Amsterdam, Sci Pk 904, NL-1098 XH Amsterdam, Netherlands
[2] Univ Stavanger, Kristine Bonnevies Vei 22, N-4021 Stavanger, Norway
来源：
ACM JOURNAL OF DATA AND INFORMATION QUALITY | 2018年 / 10卷 / 03期
关键词：
Living labs; online evaluation;
D O I：
10.1145/3239575
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
We report on our experience with TREC OpenSearch, an online evaluation campaign that enabled researchers to evaluate their experimental retrieval methods using real users of a live website. Specifically, we focus on the task of ad hoc document retrieval within the academic search domain, and work with two search engines, CiteSeerX and SSOAR, that provide us with traffic. We describe our experimental platform, which is based on the living labs methodology, and report on the experimental results obtained. We also share our experiences, challenges, and the lessons learned from running this track in 2016 and 2017.
引用
收藏
页数：15
相关论文
共 50 条
  • [1] Energy lessons learned from the 2000 campaign
    Rade, LK
    PIPELINE & GAS JOURNAL, 2001, 228 (01) : 88 - 88
  • [2] Lessons learned from the 2016 election campaign
    Johnson, J.
    ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, 2017, 254
  • [3] Lessons learned from the BOR flight campaign
    Lozino-Lozinsky, G
    Timoshenko, V
    PROCEEDINGS OF THE THIRD EUROPEAN SYMPOSIUM ON AEROTHERMODYNAMICS FOR SPACE VEHICLES, 1999, 420 : 675 - 683
  • [4] The NATO Intervention in Libya: Lessons Learned from the Campaign
    White, Dean
    INTERNATIONAL PEACEKEEPING, 2014, 21 (03) : 415 - 417
  • [5] Online Lessons Learned
    Polk, Randi L.
    FRENCH REVIEW, 2015, 89 (02): : 164 - 167
  • [6] LESSONS LEARNED FROM A PROCESS EVALUATION OF AN ONLINE ORGAN DONATION EDUCATIONAL INTERVENTION
    Redmond, Nakeva
    Harker, Laura
    Bamps, Yvan
    Flemming, Shauna
    Arriola, Kimberly J.
    Perryman, Jennie P.
    Thompson, Nancy J.
    Patzer, Rachel E.
    ANNALS OF BEHAVIORAL MEDICINE, 2017, 51 : S156 - S157
  • [7] Lessons Learned in a Breastfeeding Media Campaign
    Ware, Julie L.
    Mzayek, Fawaz
    Levy, Marian
    BREASTFEEDING MEDICINE, 2016, 11 (07) : 380 - 385
  • [8] Lessons Learned from an Organizational Information Security Awareness Campaign
    Scrimgeour, Juan-Marc
    Ophoff, Jacques
    INFORMATION SECURITY EDUCATION: EDUCATION IN PROACTIVE INFORMATION SECURITY, WISE 12, 2019, 557 : 129 - 142
  • [9] The NATO Intervention in Libya. Lessons Learned from the Campaign
    Legare, Kathia
    ETUDES INTERNATIONALES, 2015, 46 (04): : 551 - 553
  • [10] LESSONS LEARNED FROM A COMMUNITY CAMPAIGN ON CHILD SAFETY IN THE NETHERLANDS
    WORTEL, E
    DEVRIES, H
    DEGEUS, GH
    FAMILY & COMMUNITY HEALTH, 1995, 18 (03) : 60 - 77
12345