The Rayleigh-Benard convection of viscoelastic fluids in a cavity is simulated using the grid-by-grid inversion method. The grid-by-grid inversion method allows the algorithms for Newtonian fluids to be applicable to viscoelastic fluids using a technique of deferred correction. Until now, the detailed characteristics of thermal convection in a cavity have not been investigated sufficiently. Employing the grid-by-grid inversion method, we have found a peculiar phenomenon specific to the viscoelastic fluids, which cannot be predicted by the linear and nonlinear stability analysis. Contrary to the cases of Newtonian fluids and viscoelastic fluids in the parameter range of exchange of stabilities, the convection cell of viscoelastic fluids in the parameter range of Hopf bifurcation changes the circulation direction at each temporal period of oscillatory convection.