We present deep 4 hr HST/WFC images in V and I of the LBDS radio galaxy 53W002, a weak and compact narrow-line galaxy recently discovered at z = 2.390. After deconvolution, the resolution is 0.2" FWHM. Its I-band structure is quite compact with 30% +/- 10% of its flux in the central less than or similar to 1 kpc, presumably from its AGN. This is surrounded by a clearly extended and rather symmetric envelope with effective radius approximately 1.1" (4-15 kpc for H-0 = 50-100, q0 = 0-0.5). In V, 53W002 is somewhat elongated at the same sky PA as its Lyalpha cloud and its VLA radio axis. We compare 52W002's rest-frame UV profile to a properly rescaled image of a nearby early-type galaxy. The much younger starburst in 53W002 has 10-20 times higher central UV surface brightness, but otherwise an r1/4-like light profile consistent with that of nearby luminous early-type galaxies. The available data are consistent with 53W002 being a genuinely young galaxy that only started forming stars less than or similar to 0.5 Gyr before z = 2.390, but has nonetheless (just) managed to develop a rather regular light profile at z = 2.390, and suggest that the dynamical collapse of this galaxy started at about the same time as its first major burst of star formation. For this galaxy, both approximately 0.5 Gyr time scales are consistent with z(form) = 2.7-4.2 (for H-0 = 50-100, q0 = 0-0.5).