Exacting medical imaging, surgical planning, and many others are very important to handle brain image segmentation. The Convolutional Neural Networks (CNN) has been developed by the efficient auto segmentation technology. In fact, the clinical outcomes are not appropriately specific and detailed. Nevertheless, the lack of sensitivity to images and lack of generality is reduced in traditionally invisible object classes. In this paper, Deep Learning Assisted Image Interactive Medical Image Segmentation (DL-IIMIS) is proposed to tackle these difficulties by including CNNs in the bounding box and scribble-based pipeline. To adapt a CNN model to one test frame, it is proposed that image fine tuning and geodesic transformations can be either unsupervised or supervised. In this frame, two applications are involved: 2-D multi-organ magnetic resonance (MR) segmentation, with only two types of training and 3-D segmentation within brain tumor center and in entire brain tumors with different MR sequences where only one MR sequence is reported. Compared with other algorithms, the proposed framework can output a better performance in brain image segmentation.