MC Framework: High-performance Distributed Framework for Standalone Data Analysis Packages over Hadoop-based Cloud

The Hadoop MapReduce is the programming model of designing the scalable distributed computing applications, that provides developers can attain automatic parallelization. However, most complex manufacturing systems are arduous and restrictive to migrate to private clouds, due to the platform incompatible and tremendous complexity of system reconstruction. For increasing the efficiency of manufacturing systems with minimum efforts on modifying source codes, a high-performance framework is designed in this paper, called Multi-users-based Cloud-Adaptor Framework (MC-Framework), which provides the simple interface to users for fairly executing requested tasks worked with traditional standalone data analysis packages in MapReduce-based private cloud environments. Moreover, this framework focuses on multiuser workloads, but the default Hadoop scheduling scheme, i.e., FIFO, would increase delay under multiuser scenarios. Hence, a new scheduling mechanism, called Job-Sharing Scheduling, is designed to explore and fairly share the jobs to machines in the private cloud. Then, we prototype an experimental virtual-metrology module of a manufacturing system as a case study to verify and analysis the proposed MC-Framework. The results of our experiments indicate that our proposed framework enormously improved the time performance compared with the original package.