Liquid Cooling in Data Centers

Semiconductor manufacturers have aggressively attacked the problem of escalating microprocessor power consumption levels. Today, server manufacturers can purchase microprocessors that currently have power consumption levels capped at 341.2 BTU/hr (100 W) maximum. However total server power levels continue to increase, with the increase in power consumption coming from the supporting chipsets, memory, and other components. In turn, full rack heat loads are very aggressively climbing as well, and this is making it increasingly difficult and cost-prohibitive for facility owners to cool these high power racks. As a result, facilities owners are turning to alternative, and more energy efficient, cooling solutions that deploy liquids in one form or another. The paper discusses the advent of the adoption of liquid-cooling in high performance computing centers. Preparation for a liquid-cooled data center, retrofit and green field (new), is discussed herein, with a focus on the key issues that are common to all liquid-cooling technologies that depend upon the delivery of water to the rack (or in some deployments, a Coolant Distribution Unit). The paper then discusses, in some detail, the actual implementation and deployment of a liquid device-level cooled (spray cooled) supercomputer at the Pacific Northwest National Laboratory Initial results from a successful 30 day compliance test show excellent hardware stability, operating system (OS) and software stack stability, application stability and performance, and an availability level that exceeded expectations at 99.94%. The liquid-cooled supercomputer achieved a peak performance of 9.287 Tera-Flops, which placed it at number 101 in the June 2007 Top500 fastest supercomputers worldwide. Long-term performance and energy efficiency testing is currently underway, and detailed results will be reported in up coming publications.