DISTIL: A Distributed In-Memory Data Processing System for Location-Based Services

Location-based services (LBS) have become an ubiquitous technology and spatio-temporal data generated by LBS is characterized by high volume and velocity. In recent times several projects, such as GeoSpark, SpatialSpark and LocationSpark, have focused on developing spatial data systems that take advantage of the distributed in-memory data processing capability of Spark. However, most of these systems assume immutable spatial data, and they do not support high throughput location data updates that are common in LBS. On the other hand, a few HBase-based systems, such as MD-HBase, have been proposed that support data updates. However, these systems do not take advantage of any distributed in-memory query processing frameworks. To address the challenges of high velocity location data, we propose DISTIL, a distributed in-memory spatio-temporal data processing system. Our system includes a distributed in-memory index and storage infrastructure that are built on a distributed in-memory programming paradigm called APGAS (Asynchronous Partitioned Global Address Space). In our system, the location records are distributed across a cluster of nodes, using the producer-consumer model. Our experimental evaluation demonstrates that DISTIL can support high throughput location updates, and low latency concurrent processing of spatio-temporal range queries.