On-Demand Processing for Remote Sensing Big Data Analysis

In the recent decades, remote sensing data are rapidly growing in size and variety, and considered as "big geo data" because of their huge data volume, significant heterogeneity and challenge of fast analysis. In the traditional remote sensing analysis workflows, the data transfer for downloading raw image files to local workstations often costs a lot of time and slows down the data analysis workflows. Because results of remote sensing data analysis models are usually much smaller than raw data to be processed, "on-demand processing", which tries to upload data analysis models and execute them "near" where data stores, can significantly accelerate the execution of remote sensing analysis workflows. In this paper, a framework for on-demand remote sensing data analysis is proposed based on three-layered architecture; XML/JSON based runtime environment description; and on-demand model deployment methods. The evaluation on a prototype system shows that on-demand processing framework accelerates the execution of analysis models in 2.8 similar to 12.7 times by reducing data transfers, especially for those analysis workflows which transfer data through low bandwidth Internet. By on-demand processing, classical remote sensing data service systems can evolve into remote sensing data processing infrastructures, which provide IaaS (Infrastructure-as-a-Service) and PaaS (Platform-as-a Service) services, and make it possible to exchange knowledge among scientists by sharing models. Furthermore, a remote sensing data analysis platform for carbon satellites is designed based on the on-demand processing proposed by this paper and will soon be implemented under the support of SunWay-TaihuLight, the world's most powerful super computer.