Joint Visual and Textual Mining on Social Media

In modern social media, massive visual and textual data are collected and uploaded to social websites everyday. How to extract useful knowledge from such multiple modality data and organize it in an efficient way remains an important problem. The goal of this dissertation is to investigate joint visual and textual mining for social media data. My dissertation aims at contributing to our theoretical understanding on weakly supervised learning, as well as systematically building a visual and textual knowledge base. This research focuses on three phases of investigation: 1) how textual data like tags help weakly supervised visual parsing; 2) how to build a large scale knowledge base by mapping visual and textual concepts on social media; 3) with such a knowledge base, how can we interpret/organize social media data in a more reliable and effective way.