Perceptual Similarity Ranking of Temporal Heatmaps Using Convolutional Neural Networks

Similarity ranking is central to various analytic tasks. While current approaches work well on low-dimensional datasets, it becomes difficult to define similarity for more complex data types, like event sequences with multidimensional attributes. Often, the definition of similarity needs to be manually tuned according to the target domain or dataset. Visualizations are similarly manually tuned by analysts and can contain important clues about relevant features. In this paper, we propose using computer vision techniques on visualizations as a means for similarity ranking. We visualize sequential datasets as temporal heatmaps and show through user studies with 132 participants that humans agree in ranking results to a query based on perceptual similarity. We design and implement Heat2Vec, a convolutional neural network (CNN) to learn latent representations from heatmaps using color, opacity, and position. We evaluate our method against 11 baselines using a wide range of techniques and show that Heat2Vec provides rankings that are most consistently in line with human-annotated similarity ranking.