Print 10 most frequently occurring words of a text that including and excluding stopwords

python nltk word-frequency find-occurrences
user2064809 picture user2064809 · Feb 8, 2015 · Viewed 26.7k times · Source

I got the question from here with my changes. I have following code:

from nltk.corpus import stopwords
>>> def content_text(text):
    stopwords = nltk.corpus.stopwords.words('english')
    content = [w for w in text if w.lower() in stopwords]
    return content

How can I print the 10 most frequently occurring words of a text that 1)including and 2)excluding stopwords?

Answer

igorushi picture igorushi · Feb 8, 2015

There is a FreqDist function in nltk

import nltk
allWords = nltk.tokenize.word_tokenize(text)
allWordDist = nltk.FreqDist(w.lower() for w in allWords)

stopwords = nltk.corpus.stopwords.words('english')
allWordExceptStopDist = nltk.FreqDist(w.lower() for w in allWords if w not in stopwords)

to extract 10 most common:

mostCommon= allWordDist.most_common(10).keys()

Related questions

Python nltk counting word and phrase frequency

I am using NLTK and trying to get the word phrase count up to a certain length for a particular document as well as the frequency of each phrase. I tokenize the string to get the data list. from nltk.…

Read More
re.sub erroring with "Expected string or bytes-like object"

I have read multiple posts regarding this error, but I still can't figure it out. When I try to loop through my function: def fix_Plan(location): letters_only = re.sub("[^a-zA-Z]", # Search for all non-letters " ", # Replace all non-letters with …

Read More
pip issue installing almost any library

I have a difficult time using pip to install almost anything. I'm new to coding, so I thought maybe this is something I've been doing wrong and have opted out to easy_install to get most of what I needed …

Read More