Counting words in a single document from corpus in R and putting it in dataframe
I have got text documents, in each document I have text featuring tv series spoilers. Each of the documents is a different series. I want to compare the most used words of each series, I was thinking I could plot them using ggplot, and have 'Series 1 Terms that occur at least x times' on one axis and ' 'Series 2 Terms that occur at least x times' on another. I expect what I need is a dataframe with 3 columns 'Terms', 'Series x', 'Series Y'. With series x and y having the number of times that word occurs.
I have tried multiple ways to do this but failed. The closest I have got is I can read the corpus and create a dataframe with all the terms in one column like so:
library("tm")
corpus <-Corpus(DirSource("series"))
corpus.p <-tm_map(corpus, removeWords, stopwords("english")) #removes stopwords
corpus.p <-tm_map(corpus.p, stripWhitespace) #removes stopwords
corpus.p <-tm_map(corpus.p, tolower)
corpus.p <-tm_map(corpus.p, removeNumbers)
corpus.p <-tm_map(corpus.p, removePunctuation)
dtm <-DocumentTermMatrix(corpus.p)
docTermMatrix <- inspect(dtm)
termCountFrame <- data.frame(Term = colnames(docTermMatrix))
I then know I could add a column adding up the words like this:
termCountFrame$seriesX <- colSums(docTermMatrix)
but that would add occurrences from both of the documents, when I only want one.
So my questions are:
1) Is it possible to use colSums on a single doc, if not is there another way to turn the doctermmatrix into a dataframe with term counts for each document
2) Does anybody know how I can limit this so I get the most used terms in each document
Answer
If your data are in a Document Term Matrix, you'd use tm::findFreqTerms to get the most used terms in a document. Here's a reproducible example:
require(tm)
data(crude)
dtm <- DocumentTermMatrix(crude)
dtm
A document-term matrix (20 documents, 1266 terms)
Non-/sparse entries: 2255/23065
Sparsity : 91%
Maximal term length: 17
Weighting : term frequency (tf)
# find most frequent terms in all 20 docs
findFreqTerms(dtm, 2, 100)
# find the doc names
dtm$dimnames$Docs
[1] "127" "144" "191" "194" "211" "236" "237" "242" "246" "248" "273" "349" "352" "353" "368" "489" "502"
[18] "543" "704" "708"
# do freq words on one doc
findFreqTerms(dtm[dtm$dimnames$Docs == "127"], 2, 100)
[1] "crude" "cut" "diamond" "dlrs" "for" "its" "oil" "price"
[9] "prices" "reduction" "said." "that" "the" "today" "weak"
Here's how you'd find the most frequent words for each doc in the dtm, one document at a time:
# find freq words for each doc, one by one
list_freqs <- lapply(dtm$dimnames$Docs,
function(i) findFreqTerms(dtm[dtm$dimnames$Docs == i], 2, 100))
list_freqs
[[1]]
[1] "crude" "cut" "diamond" "dlrs" "for" "its" "oil" "price"
[9] "prices" "reduction" "said." "that" "the" "today" "weak"
[[2]]
[2] "\"opec" "\"the" "15.8" "ability" "above" "address" "agreement"
[8] "analysts" "and" "before" "bpd" "but" "buyers" "current"
[15] "demand" "emergency" "energy" "for" "has" "have" "higher"
[22] "hold" "industry" "its" "keep" "market" "may" "meet"
[29] "meeting" "mizrahi" "mln" "must" "next" "not" "now"
[36] "oil" "opec" "organization" "prices" "problem" "production" "said"
[43] "said." "set" "that" "the" "their" "they" "this"
[50] "through" "will"
[[3]]
[3] "canada" "canadian" "crude" "for" "oil" "price" "texaco" "the"
[[4]]
[4] "bbl." "crude" "dlrs" "for" "price" "reduced" "texas" "the" "west"
[[5]]
[5] "and" "discounted" "estimates" "for" "mln" "net" "pct" "present"
[9] "reserves" "revenues" "said" "study" "that" "the" "trust" "value"
[[6]]
[6] "ability" "above" "ali" "and" "are" "barrel."
[7] "because" "below" "bpd" "bpd." "but" "daily"
[13] "difficulties" "dlrs" "dollars" "expected" "for" "had"
[19] "has" "international" "its" "kuwait" "last" "local"
[25] "march" "markets" "meeting" "minister" "mln" "month"
[31] "official" "oil" "opec" "opec\"s" "prices" "producing"
[37] "pumping" "qatar," "quota" "referring" "said" "said."
[43] "sheikh" "such" "than" "that" "the" "their"
[49] "they" "this" "was" "were" "which" "will"
[[7]]
[7] "\"this" "and" "appears" "are" "areas" "bank"
[7] "bankers" "been" "but" "crossroads" "crucial" "economic"
[13] "economy" "embassy" "fall" "for" "general" "government"
[19] "growth" "has" "have" "indonesia\"s" "indonesia," "international"
[25] "its" "last" "measures" "nearing" "new" "oil"
[31] "over" "rate" "reduced" "report" "say" "says"
[37] "says." "sector" "since" "the" "u.s." "was"
[43] "which" "with" "world"
[[8]]
[8] "after" "and" "deposits" "had" "oil" "opec" "pct" "quotes"
[9] "riyal" "said" "the" "were" "yesterday."
[[9]]
[9] "1985/86" "1986/87" "1987/88" "abdul-aziz" "about" "and" "been"
[8] "billion" "budget" "deficit" "expenditure" "fiscal" "for" "government"
[15] "had" "its" "last" "limit" "oil" "projected" "public"
[22] "qatar," "revenue" "riyals" "riyals." "said" "sheikh" "shortfall"
[29] "that" "the" "was" "would" "year" "year's"
[[10]]
[10] "15.8" "about" "above" "accord" "agency" "ali" "among" "and"
[9] "arabia" "are" "dlrs" "for" "free" "its" "kuwait" "market"
[17] "market," "minister," "mln" "nazer" "oil" "opec" "prices" "producing"
[25] "quoted" "recent" "said" "said." "saudi" "sheikh" "spa" "stick"
[33] "that" "the" "they" "under" "was" "which" "with"
[[11]]
[11] "1.2" "and" "appeared" "arabia's" "average" "barrel." "because" "below"
[9] "bpd" "but" "corp" "crude" "december" "dlrs" "export" "exports"
[17] "february" "fell" "for" "four" "from" "gulf" "january" "january,"
[25] "last" "mln" "month" "month," "neutral" "official" "oil" "opec"
[33] "output" "prices" "production" "refinery" "said" "said." "saudi" "sell"
[41] "sources" "than" "the" "they" "throughput" "week" "yanbu" "zone"
[[12]]
[12] "and" "arab" "crude" "emirates" "gulf" "ministers" "official" "oil"
[9] "states" "the" "wam"
[[13]]
[13] "accord" "agency" "and" "arabia" "its" "nazer" "oil" "opec" "prices" "saudi" "the"
[12] "under"
[[14]]
[14] "crude" "daily" "for" "its" "oil" "opec" "pumping" "that" "the" "was"
[[15]]
[15] "after" "closed" "new" "nuclear" "oil" "plant" "port" "power" "said" "ship"
[11] "the" "was" "when"
[[16]]
[16] "about" "and" "development" "exploration" "for" "from" "help"
[8] "its" "mln" "oil" "one" "present" "prices" "research"
[15] "reserve" "said" "strategic" "the" "u.s." "with" "would"
[[17]]
[17] "about" "and" "benefits" "development" "exploration" "for" "from"
[8] "group" "help" "its" "mln" "oil" "one" "policy"
[15] "present" "prices" "protect" "research" "reserve" "said" "strategic"
[22] "study" "such" "the" "u.s." "with" "would"
[[18]]
[18] "1.50" "company" "crude" "dlrs" "for" "its" "lowered" "oil" "posted" "prices"
[11] "said" "said." "the" "union" "west"
[[19]]
[19] "according" "and" "april" "before" "can" "change" "efp"
[8] "energy" "entering" "exchange" "for" "futures" "has" "hold"
[15] "increase" "into" "mckiernan" "new" "not" "nymex" "oil"
[22] "one" "position" "prices" "rule" "said" "spokeswoman." "that"
[29] "the" "traders" "transaction" "when" "will"
[[20]]
[20] "1986," "1987" "billion" "cubic" "fiscales" "january" "mln"
[8] "pct" "petroliferos" "yacimientos"
If you want this output in a dataframe, you can do this:
# from here http://stackoverflow.com/a/7196565/1036500
L <- list_freqs
cfun <- function(L) {
pad.na <- function(x,len) {
c(x,rep(NA,len-length(x)))
}
maxlen <- max(sapply(L,length))
do.call(data.frame,lapply(L,pad.na,len=maxlen))
}
# make dataframe of words (but probably you want words as rownames and cells with counts?)
tab_freqa <- cfun(L)
But if you want to plot 'doc 1 high freq terms vs doc 2 high freq terms', then we'll need a different approach...
# convert dtm to matrix
mat <- as.matrix(dtm)
# make data frame similar to "3 columns 'Terms',
# 'Series x', 'Series Y'. With series x and y
# having the number of times that word occurs"
cb <- data.frame(doc1 = mat['127',], doc2 = mat['144',])
# keep only words that are in at least one doc
cb <- cb[rowSums(cb) > 0, ]
# plot
require(ggplot2)
ggplot(cb, aes(doc1, doc2)) +
geom_text(label = rownames(cb),
position=position_jitter())
Or perhaps slightly more efficiently, we can make one big dataframe of all the docs and make plots from that:
# this is the typical method to turn a
# dtm into a df...
df <- as.data.frame(as.matrix(dtm))
# and transpose for plotting
df <- data.frame(t(df))
# plot
require(ggplot2)
ggplot(df, aes(X127, X144)) +
geom_text(label = rownames(df),
position=position_jitter())
After you remove stopwords this will look better, but this is a good proof of concept. Is that what you were after?
