We present the first diffraction-limited K-band image of the Red Rectangle with 76 mas resolution, an H-band image with 75 mas resolution, and an RG 715 filter image ( similar to 800 nm wavelength) with 78 mas resolution (corresponding to 25 AU for a distance of 330 pc). The H and K images were reconstructed from 6 m telescope speckle data and the RG 715 image from 2.2 m telescope data using the speckle masking bispectrum method. At all wavelengths the images show a compact, highly symmetric bipolar nebula, suggesting a toroidal density distribution of the circumstellar material. No direct light from the central binary can be seen as it is obscured by a dust disk or circumbinary torus. Our first high-resolution H - K color image of the nebula shows a broad red plateau of H - K approximate to 2(m) in the bright inner regions. The optical and near-infrared images and the available photometric continuum observations in a wide range of ultraviolet to centimeter wavelengths enabled us to model the Red Rectangle in detail using a two- dimensional radiative transfer code. Our model matches both the high-resolution images and the spectral energy distribution of this object very well, making the following picture much more certain. The central close binary system with a total luminosity of 3000 L. is embedded in a very dense, compact circumbinary torus which has an average number density [n(H)] approximate to 5 x 10(12) cm(-3), an outer radius of the dense inner region of R approximate to 30 AU (91 mas), and a rho proportional to r(-2) density distribution. The full opening angle of the bipolar outflow cavities in our model is 70 degrees. By comparing the observed and theoretical images, we derived an inclination angle of the torus to the line of sight of 7 degrees +/- 1 degrees. The radiative transfer calculations show that the dust properties in the Red Rectangle are spatially inhomogeneous. The modeling confirms that the idea of large grains in the long-lived disk around the Red Rectangle (Jura et al., 1997 [ApJ, 474, 741]) is quantitatively consistent with the observations. In our models, unusually large, approximately millimeter-sized grains dominate the emission of the compact, massive torus. Models with smaller average grain sizes can possibly be found in future studies, for instance, if it turns out that the radio spectrum is not mainly caused by continuum dust emission. Therefore, the large grains suggested by our models require further confirmation by both new observations and radiative transfer calculations. Assuming a dust-to-gas ratio rho(d)/rho(g) of 0.005, the dense torus mass is 0.25 M.. The model gives a lower limit of 0.0018 M. for the mass of the large particles, which produce a gray extinction of A approximate to 28(m), towards the center. A much smaller mass of submicron-sized dust grains is presumably located in the polar outflow cavities, their conical surface layers, and in the outer low-density parts of the torus (where rho proportional to r(-4), in the region of 30 AU less than or similar to r less than or similar to 2000 AU corresponding to 0." 09- 6 ") . (C) 1998 Elsevier Science B. V. All rights reserved.