Breast cancer is the most widespread cancer among women. Based on the International cancer research center analysis, the highest number of deaths among women is due to breast cancer. Hence, detecting breast cancer at the earliest may help the oncologist to make appropriate decisions. Due to variations in breast tissue density, there is still a challenge in precise diagnosis and classification. To overcome this challenge, a novel optimal trained deep learning model (OTDEM)-based breast cancer segmentation and classification are proposed with the following four stages: they are, preprocessing, segmentation, feature extraction, and classification. The input image is passed to the initial stage using the Contrast Limited Adaptive Histogram Equalization (CLA-HE) filter to enhance the image. Then the preprocessed image is given to the segmentation stage for the image sub-segments by correlation-based deep joint segmentation. Following that, the features such as statistical features, improved local gradient texture pattern (LGXP), texton features, and shape-based features are derived from the segmented image. Then the derived features are fed to the ensemble model that includes a convolutional neural network (CNN), deep belief network (DBN), and bidirectional graph recurrent unit (Bi-GRU) classifier to finalize the classification outcome. Further, to enhance the performance of the ensemble model, the weight of BI-GRU is optimized via a new algorithm termed Swarm Intelligence - Pelican Optimization Algorithm (SIPOA). This ensures optimal training to make the model more appropriate in its classification process. Finally, the performance of the proposed work is validated over the traditional models concerning different performance measures.