Handwritten Text Segmentation via End-to-End Learning of Convolutional Neural Networks

We present a method that separates handwritten and machine-printed components that are mixed and overlapped in documents. Many conventional methods addressed this problem by extracting connected components (CCs) and classifying the extracted CCs into two classes. They were based on the assumption that two types of components are not overlapping each other, while we are focusing on more challenging and realistic cases where the components are often overlapping each other. For this, we propose a new method that performs pixel-level classification with a convolutional neural network. Unlike conventional neural network methods, our method works in an end-to-end manner and does not require any preprocessing steps (e.g., foreground extraction, handcrafted feature extraction, and so on). For the training of our network, we develop a cross-entropy based loss function to alleviate the class imbalance problem. Regarding the training dataset, although there are some datasets of mixed printed characters and handwritten scripts, most of them do not have overlapping cases and do not provide pixel-level annotations. Hence, we also propose a data synthesis method that generates realistic pixel-level training samples having many overlappings of printed and handwritten components. Experimental results on synthetic and real images have shown the effectiveness of the proposed method. Although the proposed network has been trained only with synthetic images, it also improves the OCR rate of real documents. Specifically, the OCR rate for machine-printed texts is increased from 0.8087 to 0.9442 by removing the overlapped handwritten scribbles by our method.