Volume 9, Number 12, September 2019

Semi-supervised Approach for Hindi Stemming

  Authors

Amit Anand, Sanjay Chatterji and Shaubhik Bhattacharya, IIIT Kalyani West Bengal, India

  Abstract

Stemming is one of the most fundamental requirement of any Natural Language Processing tasks such as Information Retrieval. In simple words, it is the process of finding stem of a given word. This paper presents an algorithm to find the stem of a word in Hindi. The proposed algorithm uses word2vec, which is a semisupervised learning algorithm, for finding the 10 most similar words from a corpus. Then a mathematical function is proposed to achieve the above mentioned task of finding stem. Significant amount of attention need to be given to Indo-Aryan languages like Hindi, Bengali, Marathi etc. in the domain of Natural Language Processing because of their highly inflectional properties. Moreover,it is very difficult to build a rule based stemmer for such highly conflated languages. The proposed algorithm does not need any annotated corpus and does not use any hardcoded rules for finding the stem. The results are verified by selecting a set of 1000 Hindi words randomly taken from a corpus and comparing the results given by the proposed algorithm and the actual results created manually.

  Keywords

Inflection, Stemming, Word2Vec, Unsupervised Machine Learning

Conference Proceedings