Image classification is the act of labeling groups of pixels or voxels of an image based on some rules. It finds applications in medical image analysis, and satellite image identification, along with others. Numerous studies are present in the literature where the classification is done after segmentation especially in medical images to extract only necessary areas and thereby classify them based on some criteria. It finds applications in the detection of disorders and detailed study of a particular human organ of interest. In this regard, it is important to know the challenges in this field, for accurate segmentation and classification of the region of interest. Recently, deep learning (DL) based methods for the same are being used because of higher performance as compared to the handcrafted features. Increased performance comes with various challenges like complexity, the requirement of a large amount of data, and so on. This study provides a comprehensive review of issues related to recent works on segmentation and classification techniques. This review also discusses the gaps in the literature not discussed so far and put a contributed viewpoint on the same along with future directions. It will also compare and relate each work with another and examine the datasets used along with the parametric metrics and the challenges in their use. The main focus of this review is an object-based classification used in medical imagery. It is estimated that this study will address the recent challenges and provides insights into the suggestions on different types of methods being used in the current decade.