Python NLTK: How to tag sentences with the simplified set of part-of-speech tags?

python tagging nltk
Ollie Glass picture Ollie Glass · Apr 26, 2011 · Viewed 17.3k times · Source

Chapter 5 of the Python NLTK book gives this example of tagging words in a sentence:

>>> text = nltk.word_tokenize("And now for something completely different")
>>> nltk.pos_tag(text)
[('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')]

nltk.pos_tag calls the default tagger, which uses a full set of tags. Later in the chapter a simplified set of tags is introduced.

How can I tag sentences with this simplified set of part-of-speech tags?

Also have I understood the tagger correctly, i.e. can I change the tag set that the tagger uses as I'm asking, or should I map the tags it returns on to the simplified set, or should I create a new tagger from a new, simply-tagged corpus?

Answer

Bridgette picture Bridgette · Nov 17, 2014

Updated, in case anyone runs across the same problem. NLTK has since upgraded to a "universal" tagset, source here. Once you've tagged your text, use map_tag to simplify the tags.

import nltk
from nltk.tag import pos_tag, map_tag

text = nltk.word_tokenize("And now for something completely different")
posTagged = pos_tag(text)
simplifiedTags = [(word, map_tag('en-ptb', 'universal', tag)) for word, tag in posTagged]
print(simplifiedTags)
# [('And', u'CONJ'), ('now', u'ADV'), ('for', u'ADP'), ('something', u'NOUN'), ('completely', u'ADV'), ('different', u'ADJ')]

Related questions

Count verbs, nouns, and other parts of speech with python's NLTK

I have multiple texts and I would like to create profiles of them based on their usage of various parts of speech, like nouns and verbs. Basially, I need to count how many times each part of speech is used. …

Read More
Union and Intersect in Django

class Tag(models.Model): name = models.CharField(maxlength=100) class Blog(models.Model): name = models.CharField(maxlength=100) tags = models.ManyToManyField(Tag) Simple models just to ask my question. I wonder how can i query blogs using tags in two different ways. …

Read More
How do I list all files of a directory?

How can I list all files of a directory in Python and add them to a list?

Read More