Do Teams Achieve Usability Goals? Evaluating Goal Achievement with Usability Goals Setting Tool

Do teams achieve important usability goals most of the time? Further, is goal achievement uniform or are practitioners more mindful of some goals than others? This paper presents an empirical study on usability goal achievement in industry projects. We used Usability Goal setting Tool (UGT), a recommender system that helps teams set, prioritize, and evaluate usability goals. The practitioner creates profiles for the product and its users. Based on these inputs, UGT helps the practitioner break down high-level usability goals into more specific goal parameters and provides recommendations, examples, and guidelines to assign weights to these parameters. UGT suggests strategies to evaluate goal parameters after the design is ready and assign them scores. UGT was used to collect data from 65 projects in the Indian software industry in which participants assigned weights and scores to the goal parameters. The 30 goal parameters suggested by UGT were found to be internally reliable, and having acceptable granularity and coverage. It was observed that goal parameter weights and scores correlated, but only moderately. Another interesting observation was that more than a third of the important goal parameters did not score well. We identify eight goal parameters that are typically high-weighted but have poor weight-score correlations. We call these "latent but important" goal parameters. Design teams will do well to pay closer attention to these goal parameters during projects.