nltk.word_tokenize() giving AttributeError: 'module' object has no attribute 'defaultdict'

nltk attributeerror defaultdict
Kantajit picture Kantajit · Jul 8, 2015 · Viewed 12.7k times · Source

I am new to nltk. I was trying some basics.

import nltk
nltk.word_tokenize("Tokenize me")

gives me this following error

Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
nltk.word_tokenize("hi im no onee")
File "C:\Python27\lib\site-packages\nltk\tokenize\__init__.py", line 101, in word_tokenize
return [token for sent in sent_tokenize(text, language)
File "C:\Python27\lib\site-packages\nltk\tokenize\__init__.py", line 85, in sent_tokenize
tokenizer = load('tokenizers/punkt/{0}.pickle'.format(language))
File "C:\Python27\lib\site-packages\nltk\data.py", line 786, in load
resource_val = pickle.load(opened_resource)
AttributeError: 'module' object has no attribute 'defaultdict'

Please someone help. Please tell me how to fix this error.

Answer

Manoj picture Manoj · Aug 14, 2015

I just checked it on my system.

Fix:

>> import nltk
>> nltk.download('all')

Then everything worked fine.

>> import nltk
>> nltk.word_tokenize("Tokenize me")
['Tokenize', 'me']

Related questions

re.sub erroring with "Expected string or bytes-like object"

I have read multiple posts regarding this error, but I still can't figure it out. When I try to loop through my function: def fix_Plan(location): letters_only = re.sub("[^a-zA-Z]", # Search for all non-letters " ", # Replace all non-letters with …

Read More
What is "entropy and information gain"?

I am reading this book (NLTK) and it is confusing. Entropy is defined as: Entropy is the sum of the probability of each label times the log probability of that same label How can I apply entropy and maximum entropy …

Read More
pip issue installing almost any library

I have a difficult time using pip to install almost anything. I'm new to coding, so I thought maybe this is something I've been doing wrong and have opted out to easy_install to get most of what I needed …

Read More