The (1 − y)Nd1−xYbx(Mg0.5Sn0.5)O3–yCa0.8Sr0.2TiO3 ceramics were prepared by the conventional solid-state method. The X-ray diffraction patterns of the Nd1−xYbx(Mg0.5Sn0.5)O3 ceramics revealed that Nd1−xYbx(Mg0.5Sn0.5)O3 is the main crystalline phase, which is accompanied by a little Nd2Sn2O7 as the second phase. An apparent density of 6.87 g/cm3, a dielectric constant (ɛr) of 19.48, a quality factor (Q × f) of 117,300 GHz, and a temperature coefficient of resonant frequency (τf) of −61 ppm/°C were obtained when the Nd0.96Yb0.04(Mg0.5Sn0.5)O3 ceramics were sintered at 1,600 °C for 4 h. The temperature coefficient of resonant frequency (τf) increased from −61 to −3 ppm/°C as y increased from 0 to 0.6 when the (1 − y)Nd0.96Yb0.04(Mg0.5Sn0.5)O3–yCa0.8Sr0.2TiO3 ceramics were sintered at 1,600 °C for 4 h. 0.4Nd0.96Yb0.04(Mg0.5Sn0.5)O3–0.6Ca0.8Sr0.2TiO3 ceramic that was sintered at 1,600 °C for 4 h had a τf of −3 ppm/°C.