Hybrid gradient descent spider monkey optimization (HGDSMO) algorithm for efficient resource scheduling for big data processing in heterogenous environment

Big Data constructed based on the advancement of distributed computing and virtualization is considered as the current emerging trends in Data Analytics. It is used for supporting potential utilization of computing resources focusing on, on-demand services and resource scalability. In particular, resource scheduling is considered as the process of resource distribution through an effective decision making process with the objective of facilitating required tasks over time. The incorporation of heterogeneous computing resources by the Big Data consumers also permits the option of reducing energy usage and enhanced resource efficiency. Further, optimal scheduling of resources is considered as an NP hard problem due to the dynamic characteristics of the resources and fluctuating users’ demand. In this paper, a Hybrid Gradient Descent Spider Monkey Optimization (HGDSMO) algorithm is proposed to efficient resource scheduling by handling the issues and challenges in the Hadoop heterogenous environment. The proposed HGDSMO algorithm uses the Gradient Descentand foraging and social behavior of the spider monkey optimization algorithm involved in the objective of effective resource allocation. It is designed as the efficient task scheduling approach that balances the load of the cloud by allocating them to appropriate VMs depending on their requirements. It is also proposed as a dynamic resource management scheme for efficiently allocating the cloud resources for effective execution of clients’ tasks. The simulation results of the proposed HGDSMO algorithm confirmed to be potent in throughput, load balancing and makespan compared to the baseline hybrid meta-heuristic resource allocation algorithms used for investigation.