TFIDF for Large Dataset

python lucene nlp scikit-learn tf-idf
apurva.nandan picture apurva.nandan · Aug 5, 2014 · Viewed 20.6k times · Source

I have a corpus which has around 8 million news articles, I need to get the TFIDF representation of them as a sparse matrix. I have been able to do that using scikit-learn for relatively lower number of samples, but I believe it can't be used for such a huge dataset as it loads the input matrix into memory first and that's an expensive process.

Does anyone know, what would be the best way to extract out the TFIDF vectors for large datasets?

Answer

Jonathan Villemaire-Krajden picture Jonathan Villemaire-Krajden · Aug 6, 2014

Gensim has an efficient tf-idf model and does not need to have everything in memory at once.

Your corpus simply needs to be an iterable, so it does not need to have the whole corpus in memory at a time.

The make_wiki script runs over Wikipedia in about 50m on a laptop according to the comments.

Related questions

Is there a pure Python Lucene?

The ruby folks have Ferret. Someone know of any similar initiative for Python? We're using PyLucene at current, but I'd like to investigate moving to pure Python searching.

Read More
How do I list all files of a directory?

How can I list all files of a directory in Python and add them to a list?

Read More
Iterating over dictionaries using 'for' loops

I am a bit puzzled by the following code: d = {'x': 1, 'y': 2, 'z': 3} for key in d: print key, 'corresponds to', d[key] What I don't understand is the key portion. How does Python recognize that it needs only to …

Read More