The stability of the coal seam drilling borehole is closely related to the performance of drilling fluid and reservoir conditions. In order to reduce the influence of drilling fluid intrusion, nanoparticles (NPs) have been added to drilling fluids for plugging and are widely used in the oil and gas industry. Coal is porous, hydrophobic, and has a negative charge, characteristics that are very different from shale and have a direct impact on the effectiveness of drilling fluids. It is necessary to find NPs that match the properties of the coal seam. In this study, water-based drilling fluids (WBD) with unmodified NPs (30 nm SiO2, 50 nm SiO2, and 30 nm Fe2O3) were formulated. The water loss and viscosity of drilling fluids were evaluated by filtration loss tests and rheological tests. The degree of water intrusion was evaluated by the water content. The pore characteristics of coal were tested by mercury intrusion porosimetry. The results show that pore size distribution was concentrated in a small pore size, the volume was reduced by 21.4%, and the specific surface area was reduced by 29.9%. The positive electric Fe2O3 NP was tightly combined with bentonite, which resulted in the effective particle size in the nanorange and plugged coal better. Fe2O3 NPs decreased water loss by 95.3%, and the water content of the WBD with NPs was reduced. Positively charged NPs adhere tightly to the coal seam, preventing water intrusion. The contact angle of the coal surface increased to 70.64 degrees, and the wettability was reversed. The dense and hydrophobic film formed by the WBD with NPs reduced further water intrusion. Fe2O3 NPs have an effective plugging effect and a positive effect on maintaining the stability of the wellbore.