Fault diagnosis plays an important role in guiding maintenance actions and prevent safety hazards. With the development of sensor and computer technology, deep learning (DL)-based fault diagnosis methods have been substantially developed. However, the inability to reliably represent and quantify uncertainties associated with the diagnostic results greatly hinders their industrial applicability. In this article, an uncertainty-aware fault diagnosis framework based on the Bayesian DL is proposed considering uncertainty quantification and calibration. To achieve explainable representations of different types of uncertainties, aleatoric uncertainty, epistemic uncertainty, and distributional uncertainty, which stem from the noise inherent in the observations, lack of knowledge, and domain shift, respectively, are jointly characterized for uncertainty quantification. Besides, to improve the quantification accuracy and obtain trustworthy diagnostic results to support subsequent maintenance, a novel calibration loss is proposed for the uncertainty calibration. The proposed method is applied to the two different bearing datasets to demonstrate its effectiveness in providing both the accurate diagnostic results and calibrated uncertainty quantification.