Forming Bigrams of words in list of sentences with Python

python list list-comprehension nltk collocation
Hypothetical Ninja picture Hypothetical Ninja · Feb 18, 2014 · Viewed 70.3k times · Source

I have a list of sentences:

text = ['cant railway station','citadel hotel',' police stn'].

I need to form bigram pairs and store them in a variable. The problem is that when I do that, I get a pair of sentences instead of words. Here is what I did: 

text2 = [[word for word in line.split()] for line in text]
bigrams = nltk.bigrams(text2)
print(bigrams)

which yields 

[(['cant', 'railway', 'station'], ['citadel', 'hotel']), (['citadel', 'hotel'], ['police', 'stn'])

Can't railway station and citadel hotel form one bigram. What I want is 

[([cant],[railway]),([railway],[station]),([citadel,hotel]), and so on...

The last word of the first sentence should not merge with the first word of second sentence. What should I do to make it work?

Answer

butch picture butch · Feb 18, 2014

Using list comprehensions and zip:

>>> text = ["this is a sentence", "so is this one"]
>>> bigrams = [b for l in text for b in zip(l.split(" ")[:-1], l.split(" ")[1:])]
>>> print(bigrams)
[('this', 'is'), ('is', 'a'), ('a', 'sentence'), ('so', 'is'), ('is', 'this'), ('this',     
'one')]

Related questions

if else in a list comprehension

I have a list l: l = [22, 13, 45, 50, 98, 69, 43, 44, 1] For numbers above 45 inclusive, I would like to add 1; and for numbers less than it, 5. I tried [x+1 for x in l if x >= 45 else x+5] But it gives me a syntax error. …

Read More
Pythonic way to print list items

I would like to know if there is a better way to print all objects in a Python list than this : myList = [Person("Foo"), Person("Bar")] print("\n".join(map(str, myList))) Foo Bar I read this way is not …

Read More
Python using enumerate inside list comprehension

Lets suppose I have a list like this: mylist = ["a","b","c","d"] To get the values printed along with their index I can use Python's enumerate function like this >>> for i,j in enumerate(mylist): ... print …

Read More