We present a flavor model with S-3 modular invariance in the framework of SU(5) grand unified theory (GUT). The S-3 modular forms of weights 2 and 4 give the quark and lepton mass matrices with a common complex parameter, the modulus t. The GUT relation of down-type quarks and charged leptons is imposed by the vacuum expectation value (VEV) of the adjoint 24-dimensional Higgs multiplet in addition to the VEVs of 5 and (5) over bar Higgs multiplets of SU(5). The observed Cabibbo-Kobayashi-Maskawa and Pontecorvo-Maki-Nakagawa-Sakata mixing parameters as well as the mass eigenvalues are reproduced properly. We discuss the leptonic charge-parity phase and the effective mass of the neutrinoless double beta decay with the sum of neutrino masses.