The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning

Humans have remarkable capacity to reason abductively and hypothesize about what lies beyond the literal content of an image. By identifying concrete visual clues scattered throughout a scene, we almost can't help but draw probable inferences beyond the literal scene based on our everyday experience and knowledge about theworld. For example, ifwe see a "20 mph" sign alongside a road, we might assume the street sits in a residential area (rather than on a highway), even if no houses are pictured. Can machines perform similar visual reasoning? We present Sherlock, an annotated corpus of 103K images for testing machine capacity for abductive reasoning beyond literal image contents. We adopt a free-viewing paradigm: participants first observe and identify salient clueswithin images (e.g., objects, actions) and then provide a plausible inference about the scene, given the clue. In total, we collect 363K (clue, inference) pairs, which form a first-of-its-kind abductive visual reasoning dataset. Using our corpus, we test three complementary axes of abductive reasoning. We evaluate the capacity ofmodels to: i) retrieve relevant inferences from a large candidate corpus; ii) localize evidence for inferences via boundingboxes, and iii) compare plausible inferences tomatchhumanjudgmentsonanewly-collecteddiagnosticcorpusof19KLikert-scalejudgments. While we find that fine-tuning CLIP-RN50 x 64 with amultitask objective outperforms strong baselines, significant headroom exists between model performance and human agreement. Data, models, and leaderboard available at http://visualabduction.com/.