A mathematical model for the natural convection flow and heat transfer of a nanofluid in an annulus enclosed by a square cylinder and a wavy wall cylinder is developed. Using vorticity-stream function formulation, we first derive governing equations in the Cartesian coordinates. Then, these equations are transformed utilizing coordinate transformations into a system of equations valid for the present physical domain. The problem is solved using the finite difference method. It is found that for higher values of the volume fraction of nanoparticles, the number of undulations of the wavy wall of the inner cylinder and Rayleigh number, the strength of streamlines significantly increases. However, the amplitude of undulations diminishes the intensity of streamlines. The isotherms are also strongly influenced by these parameters. Contrary to this, the Nusselt number at the inner and outer cylinders is remarkably increased due to the increase of the volume fraction of nanoparticles, amplitude of undulations, and Rayleigh number. For the higher volume fraction of nanoparticles and Rayleigh number, the average Nusselt number at the inner and outer cylinders is higher. The maximum and minimum values of the velocity profile increase with the higher Rayleigh number. Nevertheless, the converse scenario is observed for the larger amplitude of undulation and volume fraction of nanoparticles. The temperature near the inner cylinder noticeably decreases with the increase of the Rayleigh number, whereas it slowly reduces for higher amplitude of undulations. Above all, this investigation might be helpful for the researchers in regard to the approach of making a more complex geometry by using coordinate transformations. Furthermore, the results could provide vital information about the problems in current technological applications. Published under license by AIP Publishing.