It is important to maintain safety and ride quality for toll expressway users in Japan. However, since porous asphalt became the standard road surface, spot defects have gradually spread nationwide. To deal with the problem, this research attempted to develop a less costly but effective way of identifying surface defects. Since transverse data for rutting measurement was the only basic data available for general road profilers, first, quasi-three-dimensional (3D) profile data was successfully obtained by deleting gradient effects on the profiles in both the transverse and longitudinal directions. Among other elements, the standard deviation (SD) of the quasi-3D profile height using spot defect size was best matched for identifying spot defects, including pumping of underlying layer materials of the pavement. To improve the efficiency of detecting spot surface defects, deep learning was examined by converting the SD values into visual images. As a result, it was verified that a simplified classification with basic color information of red, green, and blue gave practically the same engineering judgement. Finally, this method of identifying irregularly emerging target defects using deep learning was validated by relearning the target visuals. A good result with high accuracy was achieved with just 150 images for each defect level. This approach may be universally applied anywhere surface profilers are used.