In this paper, we present the theoretical framework of quantum speed limits (QSLs) in terms of trajectory ensembles in phase space. This indicates that the QSL can be thought of as the summation of the connecting harmonic oscillators: the connections between the points of the system in phase space and the trajectory ensemble. Two typical models, the time-dependent harmonic oscillator and the undriven harmonic oscillator coupled to a thermal bath, are investigated by employing the theoretical framework. Our results from this perspective are in agreement with previous treatments.