The Pancharatnam phase for displaced number states of the harmonic oscillator is discussed. In particular, it is examined how a single quantum oscillator, driven by a suitable transient external force, evolves from the initial eigenstate u(m)(x) to the final, displaced number state modified with a suitable phase factor. The significance of this, usually neglected, phase factor for the solution of the relevant time-dependent Schrodinger equation and for the geometric phase accumulated in the wavefunction during the time evolution of the system is examined. The general expression for the geometric phase for a non-cyclic evolution from an initial displaced number state at time t1 to the final state at time t2 is derived and two applications, that of delta and harmonic forcing, are worked out. The special case of cyclic evolution is subsequently discussed and, in particular, the conditions leading to such an evolution are derived. The relationship to the classical notion of cyclic evolution is also examined in some detail and it is demonstrated that the general expression for the Pancharatnam phase reduces to the corresponding Berry phase. It is found that, in the case of cyclic evolution, the geometric phase for displaced number states becomes independent of the quantum number m of the initial eigenstsate, and becomes equal to the geometric phase for the coherent states. The general considerations are illustrated with the special case of the harmonic forcing function. Finally, the possibility of experimental verification, in the realm of quantum optics, is briefly considered.