Spatial-Crowd: A Big Data Framework for Efficient Data Visualization

Analyzing and visualizing large datasets generated by real-time spatio-temporal activities (e.g. vehicle mobility or large crowd movement) are a very challenging task. Recursive delays both at middleware and front end applications limit the of usefulness of the real-time analysis. In this paper, we present a framework "Spatial-Crowd'' that first handles spatial-temporal data acquisition and processing by scaling up the middleware components and its infrastructure. Then, it enables filtering, fixing, enriching and summarising the acquired dataset, readily available for client interfaces which usually are not scalable or built to manage such large datasets. This framework follows published subscriber model and allows users to subscribe to aggregated data streams instead of requesting data in real time. The framework is tested with data generated by a very large simulated dataset and performance showed a significant data reduction on the client side to enhance data visualisation.