Unstructured Text to Structured Data

python nlp structured-data
zenna picture zenna · Jul 2, 2010 · Viewed 7.3k times · Source

I am looking for references (tutorials, books, academic literature) concerning structuring unstructured text in a manner similar to the google calendar quick add button.

I understand this may come under the NLP category, but I am interested only in the process of going from something like "Levi jeans size 32 A0b293"

to: Brand: Levi, Size: 32, Category: Jeans, code: A0b293

I imagine it would be some combination of lexical parsing and machine learning techniques.

I am rather language agnostic but if pushed would prefer python, Matlab or C++ references

Thanks

Answer

Tal Weiss picture Tal Weiss · Jul 2, 2010

You need to provide more information about the source of the text (the web? user input?), the domain (is it just clothes?), the potential formatting and vocabulary...

Assuming worst case scenario you need to start learning NLP. A very good free book is the documentation of NLTK: http://www.nltk.org/book . It is also a very good introduction to Python and the SW is free (for various usages). Be warned: NLP is hard. It doesn't always work. It is not fun at times. The state of the art is no where near where you imagine it is.

Assuming a better scenario (your text is semi-structured) - a good free tool is pyparsing. There is a book, plenty of examples and the resulting code is extremely attractive.

I hope this helps...

Related questions

How to get rid of punctuation using NLTK tokenizer?

I'm just starting to use NLTK and I don't quite understand how to get a list of words from text. If I use nltk.word_tokenize(), I get a list of words and punctuation. I need only the words instead. …

Read More
How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?

I'm working in a sentiment analysis problem the data looks like this: label instances 5 1190 4 838 3 239 1 204 2 127 So my data is unbalanced since 1190 instances are labeled with 5. For the classification Im using scikit's SVC. The problem is I do not know how to …

Read More
Stopword removal with NLTK

I am trying to process a user entered text by removing stopwords using nltk toolkit, but with stopword-removal the words like 'and', 'or', 'not' gets removed. I want these words to be present after stopword removal process as they are …

Read More