NLTK collocations for specific words

python nltk collocation
Sabba picture Sabba · Jan 16, 2014 · Viewed 11.1k times · Source

I know how to get bigram and trigram collocations using NLTK and I apply them to my own corpora. The code is below.

I'm not sure however about (1) how to get the collocations for a particular word? (2) does NLTK have a collocation metric based on Log-Likelihood Ratio?

import nltk
from nltk.collocations import *
from nltk.tokenize import word_tokenize

text = "this is a foo bar bar black sheep  foo bar bar black sheep foo bar bar black  sheep shep bar bar black sentence"

trigram_measures = nltk.collocations.TrigramAssocMeasures()
finder = TrigramCollocationFinder.from_words(word_tokenize(text))

for i in finder.score_ngrams(trigram_measures.pmi):
    print i

Answer

bogs picture bogs · Jan 17, 2014

Try this code:

import nltk
from nltk.collocations import *
bigram_measures = nltk.collocations.BigramAssocMeasures()
trigram_measures = nltk.collocations.TrigramAssocMeasures()

# Ngrams with 'creature' as a member
creature_filter = lambda *w: 'creature' not in w


## Bigrams
finder = BigramCollocationFinder.from_words(
   nltk.corpus.genesis.words('english-web.txt'))
# only bigrams that appear 3+ times
finder.apply_freq_filter(3)
# only bigrams that contain 'creature'
finder.apply_ngram_filter(creature_filter)
# return the 10 n-grams with the highest PMI
print finder.nbest(bigram_measures.likelihood_ratio, 10)


## Trigrams
finder = TrigramCollocationFinder.from_words(
   nltk.corpus.genesis.words('english-web.txt'))
# only trigrams that appear 3+ times
finder.apply_freq_filter(3)
# only trigrams that contain 'creature'
finder.apply_ngram_filter(creature_filter)
# return the 10 n-grams with the highest PMI
print finder.nbest(trigram_measures.likelihood_ratio, 10)

It uses the likelihood measure and also filters out Ngrams that don't contain the word 'creature'

Related questions

Forming Bigrams of words in list of sentences with Python

I have a list of sentences: text = ['cant railway station','citadel hotel',' police stn']. I need to form bigram pairs and store them in a variable. The problem is that when I do that, I get a pair of …

Read More
How to get PMI scores for trigrams with NLTK Collocations? python

I know how to get bigram and trigram collocations using NLTK and I apply them to my own corpora. The code is below. My only problem is how to print out the birgram with the PMI value? I search NLTK …

Read More
How to get n-gram collocations and association in python nltk?

In this documentation, there is example using nltk.collocations.BigramAssocMeasures(), BigramCollocationFinder,nltk.collocations.TrigramAssocMeasures(), and TrigramCollocationFinder. There is example method find nbest based on pmi for bigram and trigram. example: finder = BigramCollocationFinder.from_words( ... nltk.corpus.genesis.words('english-web.txt')) &…

Read More