The sea surface temperature (SST) in the subpolar North Atlantic decreased during the past century, a remarkable feature known as the "warming hole (WH)." It is commonly claimed that the WH results from the slowdown of the Atlantic meridional overturning circulation. However, using an ensemble of the Community Earth System Model coupled to a slab ocean model simulation, we show the atmosphere alone can account for similar to 50% of the observed cooling trend in the subpolar North Atlantic and similar to 90% of the cooling relative to the temperature change of the global ocean. We find this cooling is caused by increased local westerlies in response to external forcing that enhance heat loss from the ocean through turbulent heat fluxes. This cooling is partly compensated by the damping in SST and the warming due to the rise of greenhouse gases. We further suggest that wind-driven ocean processes can enhance the cooling in the real world, but that nonetheless the ultimate driver resides in the atmosphere.