Beyond Herding Cats: Aligning Quantitative Technology Evaluation in Large-Scale Research Projects

A large-scale research project involving many research and industry organizations working on a common goal should be an ideal basis for profound technology evaluations. The possibility for industrial case studies in multiple settings ought to enable reliable quantitative assessment of the performance of new technologies in various real-world settings. However, due to diverse challenges, such as internal agendas, implicit constraints, and unaligned objectives, leveraging this potential goes beyond the usual challenge of catherding in such projects. Based on our experience from coordinating technology evaluations in several research projects, this paper sketches the typical issues and outlines an approach for dealing with them. Although new in its composition, this approach brings together principles and techniques perceived to have been useful in earlier projects (e.g., cross-organizational alignment, abstract measures, and internal baselining). Moreover, as we are currently applying the approach in a large research project, this paper presents first insights into its applicability and usefulness.