How do I tokenize a string sentence in NLTK?

python nlp tokenize nltk
diegoaguilar picture diegoaguilar · Feb 25, 2013 · Viewed 132.5k times · Source

I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I've just got up to the method like

my_text = ['This', 'is', 'my', 'text']

I'd like to discover any way to input my "text" as:

my_text = "This is my text, this is a nice way to input text."

Which method, python's or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?

Answer

Pavel Anossov picture Pavel Anossov · Feb 25, 2013

This is actually on the main page of nltk.org:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Related questions

How to get rid of punctuation using NLTK tokenizer?

I'm just starting to use NLTK and I don't quite understand how to get a list of words from text. If I use nltk.word_tokenize(), I get a list of words and punctuation. I need only the words instead. …

Read More
Python re.split() vs nltk word_tokenize and sent_tokenize

I was going through this question. Am just wondering whether NLTK would be faster than regex in word/sentence tokenization.

Read More
Python - RegEx for splitting text into sentences (sentence-tokenizing)

I want to make a list of sentences from a string and then print them out. I don't want to use NLTK to do this. So it needs to split on a period at the end of the sentence and …

Read More