Shop by image: characterizing visual search in e-commerce

Visual search has become more popular in recent years, allowing users to search by an image they are taking using their mobile device or uploading from their photo library. One domain in which visual search is especially valuable is electronic commerce, where users seek for items to purchase. Despite the increasing popularity of visual search in e-commerce, no comprehensive study has inspected its characteristics compared to traditional search using a text query. In this work, we present an in-depth comprehensive study of visual e-commerce search. We perform query log analysis of one of the largest e-commerce platforms' mobile search application. We compare visual and textual search by a variety of characteristics, with special focus on the retrieved results and user interaction with them. We also examine image query characteristics, refinement by attributes, and segmentation by user types. Additionally, we examine, for the first time, a wide variety of visual pre- and post-retrieval query performance predictors, several of which showing strong results. Our study points out a variety of differences between visual and textual e-commerce search. We discuss the implications of these differences for the design of future e-commerce search systems.