Find the tf-idf score of specific words in documents using sklearn

python scikit-learn tf-idf
WhiteTiger picture WhiteTiger · Jun 22, 2015 · Viewed 9.8k times · Source

I have code that runs basic TF-IDF vectorizer on a collection of documents, returning a sparse matrix of D X F where D is the number of documents and F is the number of terms. No problem.

But how do I find the TF-IDF score of a specific term in the document? i.e. is there some sort of dictionary between terms (in their textual representation) and their position in the resulting sparse matrix?

Answer

Ryan picture Ryan · Jun 22, 2015

Yes. See .vocabulary_ on your fitted/transformed TF-IDF vectorizer.

In [1]: from sklearn.datasets import fetch_20newsgroups

In [2]: data = fetch_20newsgroups(categories=['rec.autos'])

In [3]: from sklearn.feature_extraction.text import TfidfVectorizer

In [4]: cv = TfidfVectorizer()

In [5]: X = cv.fit_transform(data.data)

In [6]: cv.vocabulary_

It is a dictionary of the form:

{word : column index in array}

Related questions

Can I use CountVectorizer in scikit-learn to count frequency of documents that were not used to extract the tokens?

I have been working with the CountVectorizer class in scikit-learn. I understand that if used in the manner shown below, the final output will consist of an array containing counts of features, or tokens. These tokens are extracted from a …

Read More
tf-idf feature weights using sklearn.feature_extraction.text.TfidfVectorizer

this page: http://scikit-learn.org/stable/modules/feature_extraction.html mentions: As tf–idf is a very often used for text features, there is also another class called TfidfVectorizer that combines all the option of CountVectorizer and TfidfTransformer in a …

Read More
TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document

I'm using TfidfVectorizer from scikit-learn to do some feature extraction from text data. I have a CSV file with a Score (can be +1 or -1) and a Review (text). I pulled this data into a DataFrame so I can run …

Read More