An energy-aware scheduling algorithm for big data applications in Spark

Energy consumption is explosive increasing with the fast growth of big data applications. High carbon emissions from big data platforms have serious impacts on environment. In this paper, we propose an energy-aware scheduling algorithm for Spark (EASAS) to reduce energy consumption while satisfying the service level agreement (SLA). First, we present a new energy consumption model based on Spark framework. Then a strategy table for the relationship between tasks and executors is designed to record the execution time and energy consumption of tasks. The task scheduling in Spark is conducted and optimized based on the strategy table. The proposed strategy overcomes the defect of the default scheduling strategy FIFO and FAIR which cannot perceive energy consumption with the characteristics of energy consumption perception and dynamic optimization scheduling. Compared against FIFO and FAIR, Our EASAS effectively reduces on average about 25–40% of the total energy consumption of Spark applications under deadline constrains.