Data-Aware Resource Allocation of Linear Pipeline Applications in a Distributed Environment

The data stored on distributed platforms, such as cloud and fog environments, are typically processed by linear pipeline applications (LPAs). All of the component tasks of an LPA job should be assigned to the same computational resource, in order to avoid a high volume of data retrievals. On the other hand, the load balancing of the resources should also be taken into account. To this end, in this paper we investigate resource allocation strategies for LPA jobs in a distributed environment, where the input data requiring processing are not available on all of the resources. Two commonly used routing techniques are adapted in order to leverage data locality. The proposed data-aware routing policies are compared to their non-data-aware counterparts via simulation, under different workload conditions and data retrieval overhead factors. The performance of the routing techniques is examined from the mean response time and fairness perspectives. The simulation results provide useful insights into how the workload conditions and the data retrieval overhead affect the examined routing strategies.