Can We Detect Bias in Political Fact-Checking? Evidence from a Spanish Case Study

Political fact-checkers evaluate the truthfulness of politicians' claims. This paper contributes to an emerging scholarly debate on whether fact-checkers treat political parties differently in a systematic manner depending on their ideology (bias). We first examine the available approaches to analyze bias and then present a new approach in two steps. First, we propose a logistic regression model to analyze the outcomes of fact-checks and calculate how likely each political party will obtain a truth score. We test our model with a sample of fact-checks from Newtral, a prominent Spanish fact-checker. Our model would signal bias under two assumptions: (a) all political parties are on average equally accurate in their statements; (b) the verification method gives precise instructions and is implemented systematically. We investigate this second assumption with a series of interviews with Newtral fact-checkers. We show that standard verification protocols are loosely implemented and therefore fact-checks reflect a set of journalistic decisions, rather than a bias in the statistical sense. We call for a more rigorous definition of verification methods as a pre-requisite for an unbiased assessment of politician's claims.