Effective fault diagnosis under unknown operating conditions (also known as domains) is crucial for ensuring the reliability and performance of mechanical systems. However, fault diagnosis often suffers from class gaps and domain gaps among source domains, as well as between source domains and unknown target domains in realworld industrial scenarios. Current methods struggle to simultaneously tackle fault diagnosis with class gaps and domain gaps. To address these challenges, we propose a novel open-set domain generalization (OSDG) learning framework that integrates a data generation module (DGM) and a feature learning module (FLM). The DGM employs multi-domain mixup and a GAN with an auxiliary classifier to generate intra-class and open-set data, respectively, reducing class gaps. The FLM utilizes a dual-level weighted mechanism for dynamic adversarial learning, distinguishing between known and unknown classes during domain-invariant feature learning. Experimental verification was comprehensively conducted on well-known PU and DIRG datasets. The proposed method achieved average metrics on the PU dataset: accuracy on all samples (AA) of 0.9244, accuracy on known class samples (AK) of 0.9688, and accuracy on unknown class samples (AU) of 0.8584. On the DIRG dataset, the average metrics obtained were: AA of 0.8963, AK of 0.9103, and AU of 0.8902. These results surpass those of existing state-of-the-art OSDG methods. The findings confirm the effectiveness and superiority of our approach, offering a promising methodology for fault diagnosis involving unknown fault types under unknown operating conditions.