ExpRunA : a domain-specific approach for technology-oriented experiments

Conducting technology-oriented experiments (i.e., experiments in which treatments are applied to objects by a computer-based tool) without proper tool support is often a time-consuming and highly error-prone task. Although many techniques have been proposed to help conducting controlled experiments, none of them simultaneously addresses (1) the executable specification of experiments at a high level of abstraction; (2) automated treatment execution and automated data analysis from the experiment specification; and (3) formal guaranties of the correctness of results according to an experiment specification for technology-oriented experiments. To address these issues, we provide a Domain-Specific Modeling approach to create a Web-based tool (ExpRunA ) comprising a Domain-Specific Language named ToExpDSL , execution and analysis script generators, a supporting framework, and a running infrastructure. An experimenter uses ToExpDSL to specify an experiment using experimentation concepts. From this specification, applications corresponding to the underlying treatments are executed, execution results are collected and analyzed, and, finally, the analysis results are presented to the experimenter. We establish the consistency of such results with respect to the experiment specification by formalizing and proving key correctness properties of ExpRunA . We empirically evaluated ExpRunA with respect to automation by replicating three already published experiments; we evaluated the level of abstraction by a qualitative assessment. Our empirical evaluation shows that ToExpDSL is expressive enough to specify three technology-oriented experiments and that ExpRunA can be used to enable sound automation of execution and analysis from the specification of technology-oriented experiments at a high level of abstraction.