Leveraging Business Transformation with Machine Learning Experiments

The deployment of production-quality ML solutions, even for simple applications, requires significant software engineering effort. Often, companies do not fully understand the consequences and the business impact of ML-based systems, prior to the development of these systems. To minimize investment risks while evaluating the potential business impact of an ML system, companies can utilize continuous experimentation techniques. Based on action research, we report on the experience of developing and deploying a business-oriented ML-based dynamic pricing system in collaboration with a home shopping e-commerce company using a continuous experimentation (CE) approach. We identified a set of generic challenges in ML development that we present together with tactics and opportunities.