Social Relation Trait Discovery from Visual LifeLog Data with Facial Multi-Attribute Framework

Social relation defines the status of interactions among individuals or groups. Although people's happiness is significantly affected by the quality of social relationships, there are few studies focusing on this aspect. This motivates us to propose a method to discover potential social relation traits, the interpersonal feelings between two people, from visual lifelog data to improve the status of our relationships. We propose Facial Multi-Attribute Framework (FMAF), a flexible network that can embed different sets of multiple pre-trained Facial Single-Attribute Networks to capture various facial features such as head pose, expression, age, and gender for social trait evaluation. We adopt the architecture of Inception-Resnet-V2 to each single attribute component to utilize the flexibility of Inception model and avoid the degradation problem with the residual module. We use a Siamese network with two FMAFs to evaluate social relation traits for two main persons in an image. Our experiment on the social relation trait dataset by Zhangpeng Zhang et. al shows that our method achieves the accuracy of 77.30%, which is 4.10% higher than the state-of-the-art result (73.20%). We also develop a prototype system integrated into Facebook to analyse and visualize the chronological changes in social traits between a user with friends in daily lives via uploaded photos and video clips.