Probabilistic Named Entity Recognition for non-standard format entities using co-occurrence word embeddings

The use of short text has become widespread in social media like Twitter and Facebook. Typically, users on social media platforms adopt non-standard format terms when posting. This introduces challenges for Information Retrieval (IR) and Natural Language Processing (NLP) and standard or classical methods tend not to perform well in this domain. In this paper, we have addressed one of the challenges in IR which is Named Entity Recognition (NER). We introduce a novel probabilistic approach which targets entities occurring in an informal (non-standard) format within short text. The Probabilistic Named Entity Recognition (PNER) model identifies these entities using co-occurrence patterns. These patterns have been detected using the word co-occurrence embeddings of 278.6 million tweets. The results show an enhancement of 7% on two standard methods when used in combination with PNER. The testing dataset has been created using the standard methods in addition to street names and places taken from the Open Street Map (OSM) database.