Managing the Hidden Costs of Coordination

IT STARTED WITH 502 errors. Almost immediately a flood of user reports swamped the service's community Slack channel. A user posted "Getting 502s?" at 9:22 A.M., and within minutes 40 other users responded with the Yes and MeToo emojis. Also at 9:22 A.M., in an ops channel, an incident had been opened by an on-call engineer, and the site reliability engineers responsible for the service had been paged out. By 9:23 A.M. five responders were checking logs and dashboards. At 9:25 A.M.-less than two minutes after an initial tentative question indicated there may be an issue-the first notification was pushed out to users. This was aimed at slowing the influx of user reports from the 77,000-plus user community.