On the response time of large-scale composite web services

This paper studies the response time of a web service middleware that decomposes requests into sub-queries to different servers and then merges the results. We present a queuing model for such a fork-join system and an exact analysis for exponential server response times. We also provide accurate approximations for heavy-tailed server response times, which are a common effect in the Internet. Heavy-tailed distributions are critical since they may cause very long middleware response times, in particular in large-scale systems with many servers being involved. We show that in this case the performance can be significantly improved if the middleware does not have to wait for a few slow servers, i.e., if the merged result does not need to be absolutely complete. We discuss different choices to implement such a mechanism and quantify their impact on the middleware response time.