Coflow scheduling algorithm based density peaks clustering

Recently, there are many applications using coflow-based scheduling to improve performance in a cluster. However, most applications implement coflow-based scheduling by modifying their API. These applications also need to modify the third-party libraries. This is very complicated and inefficient. Therefore, prior work has proposed a coflow-based scheduling algorithm CODA, which can automatically identify coflows. CODA does not have to design the API for each application and it is transparent for the application. But CODA performs poorly on the unstable traffic and rapid traffic in datacenter. In this paper, we propose a coflow scheduling algorithm named CS-DP to solve this problem. First, we employ the density peak clustering algorithm to implement a fast, application-transparent coflow identifier. Then we employ MLFQ (multi-level feedback scheduling queues) for scheduling. Moreover, we add a threshold calculation module on MLFQ The key point of threshold calculation is to get the appropriate threshold by historical traffic. Thus, CS-DP can accommodate the current traffic quickly and effectively. Finally, the simulation results show CS-DP enabling communication stages to complete 1.11x (1.17 x ) and 2.6x(3.0x) faster on average(95-th percentile) compared to CODA algorithm and per-flow fairness on normal traffic. (C) 2019 Elsevier B.V. All rights reserved.