Do Not Shoot the Messenger: Effect of System Critical Feedback on User-Perceived Usability

Measuring perceived usability with questionnaires is a common practice for usability researchers and practitioners. This paper investigates whether there is any user bias towards the perceived usability of a system, when this system administers critical feedback to its users. These systems make decisions that substantially affect their users' lives, such as automated medical diagnosis, bank loan approval etc. In our study, we gathered data from three, almost identical, systems used to apply for a consumer loan and communicate the decision to the applicant. Our dataset involves a total of 332 applicants who completed the UMUX-LITE questionnaire after receiving the system decision (approved, rejected) for their loan. Results showed that participants who had their loans approved (positive system critical feedback) provided significantly higher UMUX-LITE scores compared to participants who had their loans rejected (negative system critical feedback). This finding suggests that one should pay attention when measuring perceived usability of critical feedback administering systems as it tends to be biased from the critical feedback that users received.