Cross-domain fault diagnosis methods have been widely developed to solve domain-shift diagnostic tasks with data distribution discrepancies. However, the quality of the domain-invariant features, which was extracted through a single channel, seriously limits the cross-domain diagnostic performance in the existing method. A novel multi-branch domain adaptation network (MBDAN), which incorporate multi-scale average processing into end-to-end deep learning model, is established in this paper. The universal feature extractor with three lightweight branches can capture the fault features from different domains. The domain adaptation strategy, which combines adversarial learning and MK-MMD-based distribution alignment, is built to learn high-quality domain-invariant features. Thus, the well-trained model can implement fault diagnosis on both the labeled source and unlabeled target domain. We conducted a comparison experiment using data from two experimental setups. The results show that MBDAN has a more remarkably cross-domain diagnostic performance than the state-of-the-art ones.