Mitigating demographic bias of machine learning models on social media

Social media posts have been used to predict different user behaviors and attitudes, including mental health condition, political affiliation, and vaccine hesitancy. Unfortunately, while social media platforms make APIs available for collecting user data, they also make it challenging to collect well structured demographic features about individuals who post on their platforms. This makes it difficult for researchers to assess the fairness of models they develop using these data. Researchers have begun considering approaches for determining fairness of machine learning models built using social media data. In this paper, we consider both the case when the sensitive demographic feature is available to the researcher and when it is not. After framing our specific problem and discussing the challenges, we focus on the scenario when the training data does not explicitly contain a sensitive demographic feature, but instead contains a hidden sensitive feature that can be approximated using a sensitive feature proxy. In this case, we propose an approach for determining whether a sensitive feature proxy exists in the training data and apply a fixing method to reduce the correlation between the sensitive feature proxy and the sensitive feature. To demonstrate our approach, we present two case studies using micro-linked Twitter/X data and show biases resulting from sensitive feature proxies that are present in the training data and are highly correlated to hidden sensitive features. We then show that a standard fixing approach can effectively reduce bias even if the sensitive attribute needs to be inferred by the researcher using existing reliable inference models. This is an important step toward understanding approaches for improving fairness on social media.