Was it Worth the Hassle? Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations

Evaluation is considered one of the major cornerstones of human-computer interaction (HCI). During the last decade, several studies have discussed pros and cons of lab and field evaluations. Based on these discussions, we conduct a review to explore the past decade of mobile HCI research on field and lab evaluation, investigating responses in the literature to the "is it worth the hassle?" paper from 2004. We find that while our knowledge and experience with both lab and field studies have grown considerably, there is still no definite answer to the lab versus field question. In response we suggest that the real question is not if - but when and how - to go into the field. In response we suggest moving beyond usability evaluations, and to engage with field studies that are truly in-the-wild, and longitudinal.