Lexicalized and statistical parsing of natural language text in Tamil using hybrid language models

Parsing is an important process of Natural Language Processing (NLP) and Computational Linguistics which is used to understand the syntax and semantics of a natural language (NL) sentences confined to the grammar. Parser is a computational system which processes input sentence according to the productions of the grammar, and builds one or more constituent structures which conform to the grammar. The interpretation of natural language text depends on the context also. Language models need syntax and semantic coverage for the better interpretation of natural language sentences in small and large vocabulary tasks. Though statistical parsing with trigram language models gives better performance through tri-gram probabilities and large vocabulary size, it has some disadvantages like lack of support in syntax, free ordering of words and long term relationship. Grammar based structural parsing provides solutions to some extent but it is very tedious for larger vocabulary corpus. To overcome these disadvantages, structural component is to be involved in statistical approach which results in hybrid language models like phrase and dependency structure language models. To add the structural component, balance the vocabulary size and meet the challenging features of Tamil language, Lexicalized and Statistical Parsing (LSP) is to be employed with the assistance of hybrid language models. This paper focuses on lexicalized and statistical parsing of natural language text in Tamil language with comparative analysis of phrase and dependency language models. For the development of hybrid language models, new part of speech (POS) tag set with more than 500 tags and dependency tag set with 31 tags for Tamil language have been developed which have the wider coverage. Phrase and dependency structure treebanks have been developed with 3261 Tamil sentences which cover 51026 words. Hybrid language models were developed using these treebanks, employed in LSP and evaluated against gold standards. This LSP with hybrid language models provides better results and covers all the challenging features of Tamil language.