Kaleidoscope: Semantically-grounded, context-specific ML model evaluation

Desired model behavior often difers across contexts (e.g., diferent geographies, communities, or institutions), but there is little infrasnotions tructure to facilitate context-specifc evaluations key to deployment decisions and building trust. Here, we present Kaleidoscope, a sysfor evaluating models in terms of user-driven, domain-relevant concepts. Kaleidoscope's iterative workfow enables generalizing from a few examples into a larger, diverse set representing an imsubreddits portant concept. These example sets can be used to test model or shifts in model behavior in semantically-meaningful ways. For instance, we might construct a "xenophobic comments" and test that its examples are more likely to be fagged by a content moderation model than a "civil discussion" set. To evalumoderators ate Kaleidoscope, we compare it against template- and DSL-based grouping methods, and conduct a usability study with 13 Reddit users testing a content moderation model. We fnd that Kaleidoscope facilitates iterative, exploratory hypothesis testing across conceptually-meaningful example sets.