This study proposed a fiber Bragg grating optical add-drop multiplexer (FBG-OADM) that is easy to produce, small, inexpensive, and conforms to International Telecommunication Union (ITU) regulations (i.e., full width at half maximum, FWHM < 0.4 nm). According to optical mode-coupled theory and the operating principle of the novel FBG-OADM, modal analysis and calculations were necessary for designing and developing a high-performance optical add-drop multiplexer (OADM). Therefore, this study adopted an improved finite element method (FEM) to thoroughly investigate and analyze the guided modes existing in the proposed FBG-OADM, thereby providing the core concepts required to design the FBG-OADM. Specifically, the modal analysis involved solving all the guided modes in the FBG-OADM, plotting 2D and 3D energy distribution graphs for the guided modes, and calculating the orthogonality and dispersion relations between the guided modes. The simulation results reveal that the higher is the order of the mode, the more sensitive it is to wavelength variations. In other words, if the novel high-performance FBG-OADM is designed with a higher mode order, then it generates smaller FWHM, thereby conforming more closely to the ITU regulation for dense wavelength-division multiplexer systems.