An Adaptive Auto-Configuration Tool for Hadoop

With the coming concept of 'big data', the ability to handle large datasets has become a critical consideration for the success of industrial organizations such as Google, Amazon, Yahoo! and Facebook. As an important Cloud Computing framework for bulk data processing, Hadoop is widely used in these organizations. However, the performance of MapReduce is seriously limited by its stiff configuration strategy. Even for a single simple job in Hadoop, a large number of tuning parameters have to be set by users. This may easily lead to performance loss due to some misconfigurations. In this paper, we present an adaptive automatic configuration tool (AACT) for Hadoop to achieve performance optimization. To achieve this goal, we propose a mathematical model which will accurately learn the relationship between system performance and configuration parameters, then configure Hadoop system based on this mathematical model. With the help of AACT, Hadoop is able to adapt the hardware and software configurations dynamically and drive the system to an optimal configuration in acceptable time. Experimental results show its efficiency and adaptability, and that it is ten times faster compared with default configuration.