A Feasibility Study of Open-Source Sentiment Analysis and Text Classification Systems on Disaster-Specific Social Media Data

Crisis informatics is a multi-disciplinary area of research that has taken on renewed urgency due to the COVID-19 pandemic and the runaway effects of climate change. Due to scarce resources, technology, especially augmented artificial intelligence (AI), has the potential to play a meaningful role by using information management for facilitating better crisis response. In part, this is both due to improvements in the underlying technology, as well as an increasing willingness by stakeholders to release data and systems as open-source. Yet, it is still not clear from published literature if such established systems are truly useful on real-world crisis datasets (such as acquired from Twitter) that often contain noise and inconsistencies. In this paper, we explore this agenda by conducting a set of case studies, using real social media data collected during six disasters (including Hurricane Sandy and the Boston Marathon Bombings) and made publicly available on a crisis informatics platform. We apply established, independently developed AI tools, including a resource specifically designed for the crisis domain, to explore whether they yield useful insights that could be helpful to first-responders. Our results reveal that, while such insights can be obtained with relatively low effort, some caveats and best practices do apply, and sentiment analysis results (in particular) are not always consistent.