Latency Modeling and Minimization for Large-scale Scientific Workflows in Distributed Network Environments

Large-scale e-science applications feature complex work-flows consisting of many computing modules. Mapping such workflows in distributed network environments and minimizing their latency are crucial to those applications that require fast system response and prompt user interaction. We model the time cost of each workflow component and design an efficient algorithm to compute the exact end-to-end delay of the entire workflow by explicitly accounting for the resource sharing dynamics. We further propose a workflow mapping approach to minimize the workflow latency using a recursive optimization procedure. The validity of the cost models and the accuracy of the latency computing algorithm are verified in comparison with an approximate solution, a dynamic system simulation program, and a workflow engine deployed in a real network. The performance superiority of the proposed mapping approach is illustrated by extensive simulation-based comparisons with existing algorithms.