Benchmarking Entity Linking for Question Answering over Knowledge Graphs

Entity Linking (EL) is the process of anchoring a part of a question to a node (entity) already known in a Knowledge Base (KB). Although EL has been widely studied with large documents such as webpages, there have not been studies about its impact on Question Answering (QA). In this paper, we study benchmarks for QA and how they are composed, providing insights about its suitability for a real evaluation about the state of the art in QA, specillay if we want to take into account the subtask of EL. We propose a semi-automatic method to generate an EL dataset linked to the QA task taking advantage of pre-existing QA datasets. We apply this method to benchmarking QA collections, analyze the results and release the created dataset to the research community, including a subset focused on complex EL in QA. We believe that EL effectiveness in the context of QA can be better assessed through the use of the proposed dataset.