The study aims to investigate experimentally the heat transfer, friction factor, and thermal performance of plate heat exchanger when Fe3O4-SiO2/water hybrid nanofluids works as a coolant. The Fe3O4-SiO2 nanoparticles were synthesized with the method of chemical coprecipitation and in-situ growth technique and characterized with x-ray diffraction, vibrating sample magnetometer, and Fourier transform infrared methods. The water-based Fe3O4-SiO2 hybrid nanofluids were prepared and the thermophysical properties were estimated experimentally. The heat transfer and friction factor experiments were conducted at mass flow rates ranging from 0.05 kg/s (Reynolds number of 245) to 0.1166 kg/s (Reynolds number of 936), and volume concentration range from 0.2% to 1.0%. By using 1.0 vol% of Fe3O4-SiO2/water hybrid nanofluids, the study found, in comparison with the base fluid that, the heat transfer coefficient is increased by 21.30%, the Nusselt number increased by 25.85%, and the friction factor increased by 37.59%, respectively, resulting in an increase of 18.1% in thermal performance factor. The Bayesian regularization-ANN analysis accurately predicts the heat transfer coefficient, friction factor and thermal performance factor using the experimental data. The multi-linear regression fit is used to develop the Nusselt number and friction factor based on the experimental data by using the ANN technique.