Quality sleep is one of the integral parts of the human body, which gives proper rest to the body. It directly promotes and supports maintaining good health, both physically and mentally. With the changes of social rhythm and increased pressure in professional sectors, it causes not to maintain proper sleep daily, which ultimately creates various sleep-related disorders. The first most important step for analyzing any type of sleep disorder is the proper classification of sleep states. The entire research work is conducted through four main steps: signal analysis, feature extraction, feature screening, and classification. This sleep study's main aim is to improve sleep staging by adding multiple channels of physiological signals such as electroencephalogram, electromyogram, and electrooculogram in an automated method. This proposed research work carried four different individual experiments conducted with the input of single-channel EEG, EMG, and EOG signals, finally experiment designed with combinations of all the three channels. This work's main focus is to analyze sleep staging accuracy with individual and combinations of signal recordings. Besides, this research work extracted 30 features, both linear and non-linear, which provide essential information with changes in subjects' sleep behavior. Further, we used a ReliefF feature selection algorithm to select the relevant features highly correlated with sleep stages, which helps analyze the sleep behavior characteristics. This study applied an AdaBoost algorithm with a base classifier as a random forest for five sleep states classification. We considered recordings of five sleep-disordered subjects and five healthy controlled subjects during experiment work the subject's data extracted from the ISRUC-Sleep dataset. The proposed methodology performances are evaluated using ISRUC-Sleep subgroup-I and subgroup-III recordings. The results of the model provide the highest classification accuracy of 98.40%, 98.49%, 98.31%, and 98.52% with EEG, EMG, EOG, and EEG + EMG + EOG respectively with ISRUC-Sleep subgroup-I data, similarly for ISRUC-Sleep subgroup-III data, the reported accuracy reached 97.96%, 98.67%, 98.40%, and 98.46%. The proposed machine learning model is ready to assist clinicians during sleep staging and diagnosis of different types of sleep disorders and can be managed with massive polysomnography records. © 2022, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd.