As an important kind of aggregation function, overlap functions are widely used in information fusion, data intelligence, image processing, decision science, etc. It is also used to construct new fuzzy rough set models. Moreover, fuzzy rough sets have been deeply studied and made great progress in data analysis and mining, feature selection, and other fields. However, after studying a large amount of literature, we find that the current research includes the following issues. First, existing fuzzy rough set models based on overlap functions (O-FRSs) need to be optimized (for example, the definition of fuzzy rough lower approximation in (I-O, O)-fuzzy rough sets is not flexible, which restricts the application of (I-O, O)-fuzzy rough sets). Second, the application research of O-FRSs is rarely involved, and the advantages of O-FRSs in feature selection and image processing are not presented. In this article, combining the above aspects, we extend (I-O, O)-fuzzy rough sets to (I, O)-fuzzy rough sets (IOFRS), which are applied to feature selection and image edge extraction systematically. First, (I, O)-fuzzy rough set model and fuzzy mathematical morphological operators based on IOFRS (IO-FMM operators) are proposed, and their relations and properties are sufficiently analyzed. Second, we propose and implement the IOFRS-based feature selection algorithm (IO-FS algorithm), and the results of 750 experiments show that the classification accuracy of IO-FS algorithm's results is better than others. Finally, combining the IO-FMM operators and fuzzy C-meaning (FCM) algorithm, an image edge detection algorithm (IO-FCM algorithm) is proposed and implemented, and the result of 35 experiments shows that the IO-FCM algorithm not only introduces the least noise but also extracts the complete image edge. The excellent performance of IOFRS in feature selection and image edge extraction fully demonstrates the advantages of O-FRSs.