Effects of Stemming on the term frequency?

data-mining text-processing tf-idf stop-words stemming
Ataman picture Ataman · May 5, 2012 · Viewed 7.6k times · Source

How are the term frequencies (TF), and inverse document frequency (IDF), affected by stop-word removal and stemming?

Thanks!

Answer

user1149501 picture user1149501 · Jun 2, 2012

tf is term frequency idf is inverse document frequency which is obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.

stemming effect is grouping all words which are derived from the same stem (ex: played, play,..), this grouping will increase the occurrence of this stem because frequencies are calculated using stem not words, For example, if you have 2 documents: the first one contains 'play' 2 times and 'played' 5 times, and the second document contains 'play' 3 times and 'played' 1 time if you do a search for 'play' without stemming the second document will be first because it has more occurrence of the word 'play', while if you do stemming, both words will be 'play' after stemming and the first document will be first cause it contains the stem play 7 times and the second document contains the stem play 4 times.

Concerning stopwords removal, it is found frequently in all document and isn't consider as a keyword for any of them, it will have high freq without any scene.

Related questions

TFIDF calculating confusion

I found the following code on the internet for calculating TFIDF: https://github.com/timtrueman/tf-idf/blob/master/tf-idf.py I added "1+" in the function def idf(word, documentList) so i won't get divided by 0 error: return math.log(len(…

Read More
How to find out if a sentence is a question (interrogative)?

Is there an open source Java library/algorithm for finding if a particular piece of text is a question or not? I am working on a question answering system that needs to analyze if the text input by user is …

Read More
What is the difference between linear regression and logistic regression?

When we have to predict the value of a categorical (or discrete) outcome we use logistic regression. I believe we use linear regression to also predict the value of an outcome given the input values. Then, what is the difference …

Read More