Event detection from real-time twitter streaming data using community detection algorithm

The increasing popularity of social media services has led to more and more people using Twitter. There are millions of tweets with a high amount of noisy data that propagate daily on the Internet. Twitter acts as a source of information for events and breaking news. However, it is very challenging for any person to extract useful information related to important events manually, from the end- less stream of tweets. Hence, it is desired to automate the whole process of event detection, so that important events can be identified in real-time from a stream of tweets, as early as possible, after the actual happening. Most of the existing approaches are more focussed on “What happened”. To define any event, answers of “When” and “Where” are also required. To handle emergency events, location and time parameters play a very important role. This article proposes a faster location based event detection approach without compromising accuracy, which automatically extracts separate clusters concerning local or global events from real-time streaming data. The proposed approach consists of four major steps. In the first step, a new dynamic weighting scheme named Conditional Term Frequency-Average Inverse Window Frequency (CTF-AIWF) based on TF-IDF is proposed to capture emerging keywords from the temporal dynamics of data. Next, a new clustering algorithm named Edge Significance based Louvain Algorithm (ESBLA) is proposed to group the same event keywords. This clustering helps in improving the run-time performance up to 50% while maintaining the quality performance (F1-score) comparable to the baseline models. In the third step, a new content-based location detection technique is proposed to detect the location of the event. This technique is able to handle various issues like use of informal text, short form of a text, and misspelled keywords of microblogging data. Finally, Google Map is used to visualize the events in happening locations. This step makes the decision faster regarding the detected events. For the experimentation, tweets are collected in real-time and stored in MongoDB NoSQL database for processing.