Self-adapting cloud services orchestration for fulfilling intensive sensory data-driven IoT workflows

Cloud computing has been adopted to support among others the storage and processing of complex Internet of Things (IoT) workflows handling sensory streamed time-series data. IoT workflow is often composed following a set of procedures which makes it hard to self-adapt, self-configure to react to runtime environment changes. Therefore, declarative data-driven workflow composition will provision self-learning and self-configurable workflows such as those of IoT. This paper proposes a comprehensive architecture to support end-to-end workflow management processes including declarative specification and composition, configuration deployment, orchestration, execution, adaptation, and quality enforcement. The later provision runtime intelligence for IoT workflow orchestration; this is achieved through the automated monitoring and analysis of runtime cloud resource orchestration, the monitoring of workflows tasks execution, as well as through cloud resource utilization prediction and workflow adaptation. In addition, it supports other intelligent features that include: (1) integration of edge computing (sensor edge) for local data processing which is very crucial for life-critical IoT workflows, (2) data compression for fast data transmission, and data storage adaptation, and (3) customization of data reporting and visualization. All these features have been evaluated through a set of experiments that proved a significant gain in terms of workflow execution time, cost and optimum usage of cloud resources compared to baseline adaptation strategy. (C) 2020 Elsevier B.V. All rights reserved.