We propose a fully ab initio approach to calculate electron-phonon scattering times for excited electrons interacting with short-wavelength (intervalley) phonons in semiconductors. Our approach is based on density functional perturbation theory and on the direct integration of electronic scattering probabilities over all possible final states with no ad hoc assumptions. We apply it to the deexcitation of hot electrons in GaAs, and calculate the lifetime of the direct exciton in GaP, both in excellent agreement with experiments. Matrix elements of the electron-phonon coupling, and their dependence on the wave vector of the final state and on the phonon modes, are shown to be crucial ingredients of the evaluation of electron-phonon scattering times.