On Building Better Mousetraps and Understanding the Human Condition: Reflections on Big Data in the Social Sciences

Over the past few years, we have seen the emergence of big data: disruptive technologies that have transformed commerce, science, and many aspects of society. Despite the tremendous enthusiasm for big data, there is no shortage of detractors. This article argues that many criticisms stem from a fundamental confusion over goals: whether the desired outcome of big data use is better science or better engineering. Critics point to the rejection of traditional data collection and analysis methods, confusion between correlation and causation, and an indifference to models with explanatory power. From the perspective of advancing social science, these are valid reservations. I contend, however, that if the end goal of big data use is to engineer computational artifacts that are more effective according to well-defined metrics, then whatever improves those metrics should be exploited without prejudice. Sound scientific reasoning, while helpful, is not necessary to improve engineering. Understanding the distinction between science and engineering resolves many of the apparent controversies surrounding big data and helps to clarify the criteria by which contributions should be assessed.