Lessons Learned from Hurricane Sandy: Maintaining Service during a Disaster

George Sherwood, the CEO of Testcover.com describes how his company maintained service during Hurricane Sandy. Testcover.com provides an automated, combinatorial test design service for software engineers, and it hosts its own service for operational flexibility, security, and cost reasons. The Testcover.com service runs on two Linux servers, one a hot standby. Internet access is by DSL, with a spare router on site. They have two uninterruptible power supplies (UPSs) for the servers and network equipment. A gasoline generator provides back-up power. The UPS company provides a Windows application for monitoring and configuring the UPS. The company sets the sensitivity to low for the PCs to reduce the frequency of switching to the battery. Five and a half days into the outage, the primary-server UPS shuts down with a discharged battery. The UPS of the standby server is pressed into service, and a smaller UPS from a PC becomes a new spare for the server. Local exchange carrier (LEC) and broadband drops are reattached with rope.