Supervised N-gram Topic Model

We propose a Bayesian nonparametric topic model that represents relationships between given labels and the corresponding words/phrases, as found in supervised articles. Unlike existing supervised topic models, our proposal, supervised N-gram topic model (SNT), focuses on both the number of topics and power-law distribution in the word frequencies for topic-specific N-grams. To achieve this goal, SNT takes a Bayesian nonparametric approach to topic sampling; it assigns a topic to each token using Chinese restaurant process (CRP), and generates a word distribution jointly with the given variable in textual order, and then forms each N-gram word as a hierarchy of Pitman-Yor process (PYP) priors. CRP can help SNT to automatically estimate the appropriate number of topics, which impacts the quality of topic specific words, N-grams, and observed value distribution. Since PYP recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing approaches for N-gram language models, it can allow SNT to generate more interpretable N-grams that the alternatives. Experiments on labeled text data show that SNT is useful as a generative model for discovering more phrases that better complement human experts than existing alternatives and provide more domain specific knowledge. The results show that SNT can be applied to various tasks such as automatic annotation.