Towards Reproducible Machine Learning Research in Information Retrieval

While recent progress in the field of machine learning (ML) and information retrieval (IR) has been significant, the reproducibility of these cutting-edge results is often lacking, with many submissions failing to provide the necessary information in order to ensure subsequent reproducibility [20, 21, 32]. Despite the introduction of self-check mechanisms before submission (such as the Reproducibility Checklist [31]), criteria for evaluating reproducibility during reviewing at several major conferences [4, 11, 28], artifact review and badging framework [18], and dedicated reproducibility tracks and challenges at major IR conferences [8, 14-17], the motivation for executing reproducible research is lacking in the broader information community. We propose this tutorial as a gentle introduction to help ensure reproducible research in IR, with a specific emphasis on ML aspects of IR research.