Deadline-aware MapReduce Scheduling with Selective Speculative Execution

MapReduce is the renowned distributed and parallel programming model with an extensive support for large scale computing. However, the performance of MapReduce is currently limited by the default scheduler as it is not suitable for heterogeneous environments and it does not consider certain user constraints such as deadlines. The paper proposes a deadline aware scheduling algorithm that selectively uses speculative execution when the job approaches its deadline in order to expedite job's execution. The algorithm is implemented on the heterogeneous Hadoop cluster and the evaluation shows significant improvement in the performance. The performance improvement was observed as the number of jobs that miss the deadlines as well as overall execution time for different workloads was minimized