Federated Learning for IoMT Applications: A Standardization and Benchmarking Framework of Intrusion Detection Systems

Efficient evaluation for machine learning (ML)-based intrusion detection systems (IDSs) for federated learning (FL) in the Internet of Medical Things (IoMTs) environment falls under the standardisation and multicriteria decision-making (MCDM) problems. Thus, this study is developing an MCDM framework for standardising and benchmarking the ML-based IDSs used in the FL architecture of IoMT applications. In the methodology, firstly, the evaluation criteria of ML-based IDSs are standardised using the fuzzy Delphi method (FDM). Secondly, the evaluation decision matrix (DM) is formulated based on the intersection of standardised evaluation criteria and a list of ML-based IDSs. Such formulation is achieved using a dataset with 125,973 records, and each record comprises 41 features. Thirdly, the integration of MCDM methods is formulated to determine the importance weights of the main and sub standardised security and performance criteria, followed by benchmarking and selecting the optimal ML-based IDSs. In this phase, the Borda voting method is used to unify the different ranks and perform a group benchmarking context. The following results are confirmed. (1) Using FDM, 17 out of 20 evaluation criteria (14 for security and 3 for performance) reach the consensus of experts. (2) The area under curve criterion has the lowest set of weights, whilst the CPU time criterion has the highest one. (3) VIKOR group ranking shows that the BayesNet is a best classifier, whilst SVM is the last choice. For evaluation, three assessments, namely, systematic ranking, computational cost and comparative analysis, are used.