Using k-Way Co-Occurrences for Learning Word Embeddings

Co-occurrences between two words provide useful insights into the semantics of those words. Consequently, numerous prior work on word embedding learning has used co-occurrences between two words as the training signal for learning word embeddings. Flowever, in natural language texts it is common for multiple words to be related and cooccurring in the same context. We extend the notion of co-occurrences to cover k(>= 2)-way co-occurrences among a set of k-words. Specifically, we prove a theoretical relationship between the joint probability of k(>= 2) words, and the sum of l(2) norms of their embeddings. Next, we propose a learning objective motivated by our theoretical result that utilises k-way Co-occurrences for learning word embeddings. Our experimental results show that the derived theoretical relationship does indeed hold empirically, and despite data sparsity, for some smaller k(<= 5) values, k-way embeddings perform comparably or better than 2-way embeddings in a range of tasks.