Getting the root word using the Wordnet Lemmatizer

python nlp nltk wordnet lemmatization
Shanika Ediriweera picture Shanika Ediriweera · Sep 3, 2016 · Viewed 7.5k times · Source

I need to find a common root word matched for all related words for a keyword extractor.

How to convert words into the same root using the python nltk lemmatizer?

  • Eg:
    1. generalized, generalization -> general
    2. optimal, optimized -> optimize (maybe)
    3. configure, configuration, configured -> configure

The python nltk lemmatizer gives 'generalize', for 'generalized' and 'generalizing' when part of speech(pos) tag parameter is used but not for 'generalization'.

Is there a way to do this?

Answer

Ani Menon picture Ani Menon · Sep 3, 2016

Use SnowballStemmer:

>>> from nltk.stem.snowball import SnowballStemmer
>>> stemmer = SnowballStemmer("english")
>>> print(stemmer.stem("generalized"))
general
>>> print(stemmer.stem("generalization"))
general

Note: Lemmatisation is closely related to stemming. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech.

A general issue I have seen with lemmatizers is that it identifies even bigger words as lemmas.

Example: In WordNet Lemmatizer(checked in NLTK),

  • Genralized => Generalize
  • Generalization => Generalization
  • Generalizations => Generalization

POS tag was not given as input in the above cases, so it was always considered noun.

Related questions

WordNetLemmatizer not returning the right lemma unless POS is explicit - Python NLTK

I'm lemmatizing the Ted Dataset Transcript. There's something strange I notice: Not all words are being lemmatized. To say, selected -> select Which is right. However, involved !-> involve and horsing !-> horse unless I explicitly input …

Read More
Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?

Using NLTK and WordNet, how do I convert simple tense verb into its present, past or past participle form? For example: I want to write a function which would give me verb in expected form as follows. v = 'go' present = …

Read More
Extract Word from Synset using Wordnet in NLTK 3.0

Some time ago, someone on SO asked how to retrieve a list of words for a given synset using NLTK's wordnet wrapper. Here is one of the suggested responses: for synset in wn.synsets('dog'): print synset.lemmas[0].name Running …

Read More