We demonstrate that in confined plasmonic metal structures subject to ultrafast laser excitation, electron thermal diffusion (ETD) can provide spatial redistribution of excess energy faster than its transfer to the lattice. This relaxation occurs after the excitation of nanometer-sized hot spots in the confined structure, changing sensitively the optical parameters in these regions. The changes become essential when the plasmonic resonance condition is met for both excitation and detection, as evidenced by a pump-probe experiment on plasmonic gold lattices: Subpicosecond relaxation with characteristic times well described by a two-temperature model involving ETD is observed. The results suggest that the dynamical optical response in plasmonic structures can be tuned by the selection of the structural geometry as well as the choice of wavelength and polarization of the excitation and detection light.