Predicting Trends in the Twitter Social Network: A Machine Learning Approach

The Twitter microblogging site is one of the most popular websites in the Web today, where millions of users post real-time messages (tweets) on different topics of their interest. The content that becomes popular in Twitter (i.e., discussed by a large number of users) on a certain day can be used for a variety of purposes, including recommendation of popular content and marketing and advertisement campaigns. In this scenario, it would be of great interest to be able to predict what content will become popular topics of discussion in Twitter in the recent future. This problem is very challenging due to the inherent dynamicity in the Twitter system, where topics can become hugely popular within short intervals of time. The Twitter site periodically declares a set of trending topics, which are the keywords (e.g., hashtags) that are at the center of discussion in the Twitter network at a given point of time. However, the exact algorithm that Twitter uses to identify the trending topics at a certain time is not known publicly. In this paper, we aim to predict the keywords (hashtags) that are likely to become trending in Twitter in the recent future. We model this prediction task as a machine learning classification problem, and analyze millions of tweets from the Twitter stream to identify features for distinguishing between trending hashtags and non-trending ones. We train classifiers on features measured over one day, and use the classifiers to distinguish between trending and non-trending hashtags on the next day. The classifiers achieve very high precision and reasonably high recall in identifying the hashtags that are likely to become trending.