Online Learning Sum-Product Networks for Language Modeling

Sum-product networks (SPNs) have recently proposed as an remarkable representation due to their dual view as a special deep neural network with clear semantics and a probabilistic graphical model for which inference is always tractable. SPNs have been successfully applied in Computer Vision and Natural Language Processing. We used the hidden layers of SPNs to model complex dependencies among words and we used SPNs online learning algorithm to improve model learning speed and SPNs structure learning algorithm to improve modeling capabilities. Our empirical comparisons with other previous language models indicate that our online learning SPNs has better performance.