A Scalable Framework for Cloud Powered Workflow Execution

Executing workflows remotely in cloud can help to improve the execution speed of scientific workflows that require high computational power. Many existing frameworks for remote workflow execution are only an extension of the desktop software environment. This precludes execution of hybrid workflows using algorithms from different software. In addition, these frameworks also rely on existing cluster management infrastructure and require the clusters to be manually prepared before workflow submission. These issues limit the application of cloud based workflow execution. In this paper, we propose a scalable framework, CloudPower, for cloud powered workflow execution. This framework can support hybrid workflows and automatic cluster provisioning. It also supports automatic scaling from private cloud to public cloud. To improve the workflow execution speed, CloudPower supports several execution parallelization strategies such as parameter sweeping, data splitting. We also propose an operator parallelization algorithm to enable parallel execution of operators from the same workflow. Various design issues and several applications of the framework are also discussed.