Automated shopping system using computer vision

The shopping experiment made by amazon go in USA is one of the most interesting applications of computer vision recently. They allow you to shop and automatically charge your virtual card for whatever goods you purchased using cameras and wireless systems, so no checkouts or waiting lines are required. However, amazon didn't reveal yet the details of how their system components are implemented. In this paper, we introduce a complete system for computer vision based automated shopping. The proposed system contains barcode scanning of objects, data registration, image capturing for offline training stage, motion (change) detection, CNN and SVM for object classification and charging/discharging customers. Our system can be integrated with the wireless data transmission to do the whole shopping process. First, the proposed method extracts the objects' barcodes to register their details, and take sample images of objects for classifier training. We employ a pre-trained CNN (i.e. ResNet50) for feature extraction and a multi-class SVM for training. After training our classifier, we have a real-time operation stage (i.e. test stage). We assume that a camera is embedded above products on each shelf to capture videos of the products. We employ a change detector to understand any added or removed items. If the item is removed from or added to the shelve, the moving object is input to CNN feature extractor, and then SVM classifier for identification and pricing. Results show that the proposed system is fast and effective.