Exploring how software developers work with mention bot in GitHub

Recently, major software development platforms have started to provide automatic reviewer recommendation (ARR) services for pull requests to improve collaborative coding review process. However, the user experience of ARR is under-investigated. In this paper, we use a two-stage mixed-methods approach to study how software developers perceive and work with the Facebook mention bot, one of the most popular ARR bots in GitHub. Specifically, in Stage I, we conduct archival analysis on projects employing mention bot and a user survey to investigate bot performance. A year later, in Stage II, we revisit these projects and conduct additional surveys and interviews with three user groups: project owners, contributors and reviewers. Results show that developers appreciate mention bot saving their efforts, but are bothered by its unstable setting and unbalanced workload allocation. We conclude with design considerations for improving ARR services.