Cloud Crawler: a declarative performance evaluation environment for infrastructure-as-a-service clouds

As the number of infrastructure-as-a-service (IaaS) cloud offers in the market increases, selecting an appropriate configuration of cloud resources for a given application becomes a non-trivial task even for experienced developers. Because cloud resources are relatively cheap, usually charged by the hour, developers could systematically evaluate the performance of their application using different resource types from different cloud providers, thus allowing them to accurately identify the best providers and resource types for their application. However, conducting systematic performance tests in multiple IaaS clouds may require a significant amount of planning and configuration effort from application developers. This paper presents Cloud Crawler, a declarative environment for specifying and conducting application performance tests in IaaS clouds. The environment includes a novel declarative domain-specific language, Crawl, by means of which cloud users can describe, at a high abstraction level, a large variety of performance evaluation scenarios for a given application, and a scenario execution engine, Crawler, which automatically configures, executes, and collects the results of the scenarios described in Crawl. The paper also reports on how Cloud Crawler has been successfully used to systematically test the performance of two open-source web applications in public IaaS clouds. Copyright (c) 2016 John Wiley & Sons, Ltd.