How to calculate TF*IDF for a single new document to be classified?

Question

I am using document-term vectors to represent a collection of document. I use TF*IDF to calculate the term weight for each document vector. Then I could use this matrix to train a model for document classification.

I am looking forward to classify new document in future. But in order to classify it, I need to turn the document into a document-term vector first, and the vector should be composed of TF*IDF values, too.

My question is, how could I calculate the TF*IDF with just a single document?

As far as I understand, TF can be calculated based on a single document itself, but the IDF can only be calculated with a collection of document. In my current experiment, I actually calculate the TF*IDF value for the whole collection of documents. And then I use some documents as training set and the others as test set.

I just suddenly realized that this seems not so applicable to real life.

ADD 1

So there are actually 2 subtly different scenarios for classification:

1. to classify some documents whose content are known but label are not known.
2. to classify some totally unseen document.

For 1, we can combine all the documents, both with and without labels. And get the TF*IDF over all of them. This way, even we only use the documents with labels for training, the training result will still contain the influence of the documents without labels.

But my scenario is 2.

Suppose I have the following information for term T from the summary of the training set corpus:

• document count for T in the training set is n
• total number of training documents is N

Should I calculate the IDF of t for a unseen document D as below?

IDF(t, D)= log((N+1)/(n+1))

ADD 2

And what if I encounter a term in the new document which didn't show up in the training corpus before? How should I calculate the weight for it in the doc-term vector?

Answer 1

TF-IDF doesn't make sense for a single document, independent of a corpus. It's fundamentally about emphasizing relatively rare and informative words.

You need to keep corpus summary information in order to compute TF-IDF weights. In particular, you need the document count for each term and the total number of documents.

Whether you want to use summary information from the whole training set and test set for TF-IDF, or for just the training set is a matter of your problem formulation. If it's the case that you only care to apply your classification system to documents whose contents you have, but whose labels you do not have (this is actually pretty common), then using TF-IDF for the entire corpus is okay. If you want to apply your classification system to entirely unseen documents after you train, then you only want to use the TF-IDF summary information from the training set.