A framework to simplify pre-processing location-based social media big data for sustainable urban planning and management

Over the last decade, 90% of Big Data has been generated by people living in urban areas. With the advent of Internet of Things (IoT) and the increased use of the internet, Social Media has become an integral part of people's daily lives. Millions of unstructured data are being sent to the cloud every second, providing opinions practically on any discourse. This makes microblogs such as Twitter, Instagram, WeChat, and Facebook smart instruments for urban planners to harvest 'big data' on socioeconomics, urban dynamics, transportation, land uses, resilience, etc. This study proposed a framework for social media big data mining and data analytics using Twitter. It demonstrated the functionalities of the framework on a case study using Natural Language Processing and Machine Learning techniques like Latent Dirichlet Allocation and VADER Sentiment Analysis to mine, clean, process, and validate the data. The validated results from the case study showed high accuracy that Social Media Big Data can be used to study the spatiotemporal dynamism of community challenges.