XPNet: Cross-Domain Prototypical Network for Zero-Shot Sketch-Based Image Retrieval

Zero-shot retrieval is a topical problem for sketch-based image search. It is largely necessitated by the fact that human sketch data is scarce in nature - in most cases retrieval will have to be conducted at zero-shot level. The problem of zero-shot sketch-based image retrieval (ZS-SBIR) is however a much harder task when compared with its photoonly counterpart. In addition to addressing the zero-shot transfer problem, it will also need to tackle the inherent domain gap between sketch and photo. Most existing works on ZS-SBIR typically address these two problems separately: a triplet-like network to address the domain gap, and employing external semantic information (such as word embeddings) to assist category transfer. In this paper, we take a different stance and ask a more difficult question - can we devise a consolidated solution to accommodate both problems simultaneously, especially without the need for additional semantic information. For that, we propose a cross-domain prototype learning framework to narrow the domain gap by encouraging a confirmation of prototypes between two domains. The intuition is there exists an embedding in which points regardless of which domain it comes from, would cluster around a single and shared prototype representation for a given class. We first show that performance comparable with that of state-of-the-art can already be achieved just by doing this alone. We then further propose two means of tackling data efficiency during training: (i) an episode training protocol that enables data feeding by demand, and (ii) a hard triplet generation algorithm to address data scarcity. Extensive experiments on TU-Berlin-Extended, Sketchy-Extended and QuickDraw-Extended validate the usefulness of our approach.