Budget Distribution Strategies for Scientific Workflow Scheduling in Commercial Clouds

Scientific research is increasingly reliant on big compute and big data, the fusion of which is known as data intensive science. Large scale scientific analyses are typically represented as workflows which are the typical model for characterizing e-science experiments in distributed systems. Workflows with a large number of tasks are distributed in parallel across computing resources to speed up analyses. The provision of compute capabilities is undergoing a rapid migration from dedicated infrastructure to the cloud. This migration is fuelled by dynamic infrastructure scalability with changes in demand. Cloud instances incur different costs and execution time with different configurations. A key concern for workflow scheduling is to make an appropriate trade-off between these two factors. In this paper, we introduce the Budget Distribution with Trickling (BDT) algorithm that presents new notions for distributing budget based on the dependency structure inherent in workflows. In addition we propose several new strategies for sharing or distributing the budget, and propose trickling to redistribute unspent budget down to other levels. Our results show that biasing the budget distribution to the earlier computation within a workflow will generally produce a lower makespan within budget.