Driver decisions and behaviors are the major factors in on-road driving safety. Most significantly, traffic injuries and accidents are reduced using the accurate driver behavior monitoring system. However, the challenges occur in understanding human behaviors in the practical environment due to uncontrolled scenarios like cluttered and dynamic backgrounds, occlusion, and illumination variation. Recently, traffic accidents are mainly caused by distracted drivers, which has increased with the popularization of smartphones. Therefore, the distracted driver detection model is necessary to appropriately find the behavior of the distracted driver and give warnings to the driver to prevent accidents, which need to be concentrated as serious issues. The main intention of this paper is to design and implement a novel deep learning framework for driver distraction detection. First, the datasets for driver distraction detection are gathered from public sources. Furthermore, the Optimal Fusion-based Local Gradient Pattern (LGP) and Local Weber Pattern (LWP) perform the pattern extraction of the images. These patterns are inputted into the new deep learning framework with Ensemble Variant Convolutional Neural Network (EV-CNN) for feature learning. The EV-CNN includes three different models, like Resnet50, Inceptionv3, and Xception. The extracted features are subjected (HSWOA) performs both the pattern extraction and the LSTM optimization. The experimental results demonstrate the effective classification performance of the suggested model in terms of accuracy during the detection of distracted driving and are helpful in maintaining safe driving habits.