Coffee is the most regularly consumed beverages around the world, and the quality of coffee beans depends greatly on the health of the coffee plants. To prevent coffee leaf diseases (CLDs) from spreading and ensure a healthy coffee leaf, it is critical to detect them accurately and early. Coffee is a vital commodity for the global economy, and diseases affect its productivity and quality, so an automatic method for detecting coffee leaf disorder is necessary. In this work, a novel Coffee-Net is proposed for precisely classify and identify the diseases in CLDs, namely, Phoma, miner, rust, and Cercospora. Bilateral filter is used to smooth the image and remove additive noise, followed by contrast stretching adaptive histogram equalization (CSAHE) to enhance the quality of the image. Patches generated by region convolutional neural network (RCNN) automatically vary a transform that captures the common pattern for patch generation. The input image is then separated into multiple patches and the patch results are given for classification. The latent representation of coffee leaf can be classified into healthy, Phoma, miner, rust, and Cercospora based on the correct prediction of these images. The proposed Coffee-Net achieves 99.95% of accuracy rate for the CLD detection. The proposed Coffee-Net outperforms artificial neural network (ANN), Mask R-CNN, MobileNetV2, and ResNet50 in terms of overall accuracy by 0.6%, 4.32%, 0.02%, and 1.95, accordingly.