Permeability and its anisotropy are of central importance for groundwater and hydrocarbon migration. Existing fluid dynamics methods for computing permeability have common shortcomings, i.e., high computational complexity and long computational time, reducing the potential of these methods in practical applications. In view of this, a 3D CNN-based approach for rapidly estimating permeability in anisotropic rock is proposed. Using high-resolution X-ray microtomographic images of a sandstone sample, numerous samples of the size of 100-cube voxels were generated firstly by a series of image manipulation techniques. The shrinking and expanding algorithms are employed as the data augmentation methods to strengthen the role of porosity and specific surface area (SSA) since these two parameters are critical to estimate permeability. Afterwards, direct pore-scale modeling with Lattice-Boltzmann method (LBM) was utilized to compute the permeabilities in the direction of three coordinate axes and mean permeability as the ground truth. A dataset including 3158 samples for training and 57 samples for testing were obtained. Four 3D CNN models with the same network structure, corresponding to permeabilities in 3 directions and in average, were built and trained. Based on those trained models, the satisfactory predictions of the permeabilities in x-, y-, and z-axis directions and the mean permeability were achieved with R2 scores of 0.8972, 0.8821, 0.8201, and 0.9155, respectively. Furthermore, those proposed 3D CNN models achieved good generalization ability in predicting the permeability of other samples. The trained model takes only tens of milliseconds on average to predict the permeability of one sample in one axial direction, about 10,000 times faster than LBM. The promising performance clearly demonstrates the effectiveness of 3D CNN-based approach in rapidly estimating permeability in anisotropic rock. This new approach provides an alternative way to calculate permeability with low computing cost, and it has the potential to be extended to the estimation of relative permeability and other properties of rocks.