MixHeter: A global scheduler for mixed workloads in heterogeneous environments

As data centers and applications grow more heterogeneous, allocating the proper resources to various applications increasingly depends on understanding the trade offs between different allocations, because mixed workloads may benefit from different resources e.g. GPU, Solid State Drives(as SSD). However, traditional distributed programming models are designed and improved for homogeneous environments and have poor performances in current heterogeneous environments. Thus we reconsider the problem in this paper and make three contributions: (1) Through analysis of experimental results, we summarize the main reasons of poor performances are unreasonable allocation of tasks between heterogeneous nodes and improper allocation of resources to mixed workloads; (2) To resolve them, we propose a global scheduler MixHeter based on or-constraints. Or-constraints imbibes advantages of no-constraints and hard-constraints, which satisfy applications' resource preferences when all the resources are available and do not waste the non-preferred resources when the preferred resources are occupied. The model of or-constraints is based on utility function, which can associate utility with different resource requests to represent resource preferences and maximize overall utility to improve system efficiency. (3) Finally, we prove MixHeter can greatly decrease execution time than capacity scheduler of Hadoop 2.7.3 and capacity scheduler with label-based scheduling up to 15%-60% in heterogeneous environments, especially in the condition of mixed workloads with different resource preferences. (C) 2017 Elsevier Inc. All rights reserved.