Reading Numbers in Natural Scene Images with Convolutional Neural Networks

Reading text from natural images is a hard computer vision task. We present a method for applying deep convolutional neural networks to recognize numbers in natural scene images. In this paper, we proposed a noval method to eliminating the need of explicit segmentation when deal with multi-digit number recognition in natural scene images. Convolution Neural Network(CNN) requires fixed dimensional input while number images contain unknown amount of digits. Our method integrats CNN with probabilistic graphical model to deal with the problem. We use hidden Markov model(HMM) to model the image and use CNN to model digits appearance. This method combines the advantages of both the two models and make them fit to the problem. By using this method we can perform the training and recognition procedure both at word level. There is no explicit segmentation operation at all which save lots of labour for sophisticated segmentation algorithm design or finegrained character labeling. Experiments show that deep CNN can dramaticly improve the performance compared with using Gaussian Mixture model as the digit model. We obtaied competitive results on the street view house number(SVHN) dataset.