Storage Solution: A Virtual Distributed Storage And Migration Architecture For Big Data

The big data revolution has provided organizations with a large variety of data which assist in making decisions as well as providing data analysts with high volumes of data for prediction and pattern recognition. This data is stored in the cloud as it has proven to be a good storage environment due to its accessibility and security benefits.The cloud provides data for various applications such as gaming activities and data for prediction analysis in various sectors of the economy.The provision of these data to end-users such as data analysts is best provided through the use virtual desktops which require data regularly (real-time) and at a high speed, efficiency and performance levels. To achieve this users expect services to be hosted on virtual machines in interrelated data centres and that these virtual machines will migrate dynamically to locations best suited for the user as well as connect the new users.This leads to our current problem which is how can we provide high performance to manage large volumes of data from the cloud as well as how can data can be stored in such a manner that they can be easily retrieved and migrated between servers. We propose an Architecture in this paper by using Alluxio and our novel Dynamic Virtual Machine Server (DVMS) to speed up the process as well as ensure there is no delay. We further apply two plugins to Hadoop which are Sqoop and Network levitated Merge (NLM) which will assist to improve the transfer speed of data from the cloud to Hadoop to increase efficiency. The dynamic virtual machine manages the large and growing data load by categorising the data into 3 categories of pools called (1) Raw aggregated data pool, (2) Aggregated data to send and (3) Processed aggregated data pool which works in a loop to increase data migration speed as well as provide a medium to store data in preparation for new users.