D3CAS: Distributed Clustering Algorithm Applied to Short-Text Stream Processing

In this article, a proof of concept of a dynamic clustering algorithm based on density, called D3CAS, is presented. This algorithm was implemented to be run under the Spark Streaming framework, and it allows processing data streams. The algorithm was tested using a stream of short texts consisting of requirements generated by social media users, in particular, from a dataset called Pizza Request Dataset. The results, obtained in a virtualized environment, were analyzed with different configurations for algorithm parameters, which allowed establishing which are the configurations that yield the best results. Since the dataset used includes the label for each text in the stream, cluster purity could be measured and the results obtained could be compared to those presented by the authors of the dataset.