An Architectural Framework for Quality-driven Adaptive Continuous Experimentation

Continuous experimentation enables companies to reduce development risks and operational costs by continuously and directly assessing user response with respect to software updates. The increasing need for data-driven rapid decisions to face unpredictable context situations demands the automation of continuous experimentation practices. Furthermore, variable conditions and constraints associated with the experimentation process, such as changes in the experimentation goals and the cost of conducting experimental trials, demand from experiments to be adaptive. This paper presents our proposal towards what we call quality-driven adaptive continuous experimentation. Our contributions are as follows. First, we present a metamodel for experimental design to enable automatic planning and execution of experiments at run-time. Second, we propose a mesh of runtime models to allow autonomic managers conduct experiments while assisting in the continuous evolution of the subject system. Finally, we propose an architecture for quality-driven adaptive experimentation. Our architecture addresses separation of concerns in the experimentation process by dedicating three feedback loops to (1) control the satisfaction of high-level experimentation goals through experimental design; (2) conduct experimental trials for infrastructure configuration variants; and (3) conduct experimental trials for architectural design variants.