Semantic granularity metric learning for visual search

Existing metric learning methods often do not consider different granularly in visual similarly. However, in many domains, images exhibit similarly at multiple granularities with visual semantic concepts, e.g. fashion demonstrates similarly ranging from clothing of the exact same instance to similar looks/design or common category. Therefore, training image triplets/pairs inherently possess different degree of information. Nevertheless, the existing methods often treat them with equal importance which hinder capturing underlying granularities in image similarly. In view of this, we propose a new semantic granularly metric learning (SGML) that develops a novel idea of detecting and leveraging attribute semantic space and integrating it into deep metric learning to capture multiple granularities of similarly. The proposed framework simultaneously learns image attributes and embeddings with multitask-CNN where the tasks are linked by semantic granularly similarly mapping to leverage correlations between the tasks. To this end, we propose a new soft-binomial deviance loss that effectively integrates informativeness of training samples into metric-learning on-the-fly during training. Compared to recent ensemble-based methods, SGML is conceptually elegant, computationally simple yet effective. Extensive experiments on benchmark datasets demonstrate its superiorly e.g., 1-4.5%-Recall@1 improvement over the state-of-the-arts (Kim a al., 2018; Cakir a al., 2019) on DeepFashion-Inshop