R, tm-error of transformation drops documents

r extract keyword tm extraction
Julie picture Julie · Aug 21, 2018 · Viewed 14.6k times · Source

I want to create a network based on the weight of keywords from text. Then I got an error when running the codes related to tm_map:

library (tm)
library(NLP)
lirary (openNLP)

text = c('.......')
corp <- Corpus(VectorSource(text))
corp <- tm_map(corp, stripWhitespace)

Warning message:
In tm_map.SimpleCorpus(corp, stripWhitespace) :
transformation drops documents

corp <- tm_map(corp, tolower)

Warning message:
In tm_map.SimpleCorpus(corp, tolower) : transformation drops documents

The codes were working 2 months ago, now I'm trying for a new data and it is not working anymore. Anyone please shows me where was I wrong. Thank you. I even tried with the command below, but it doesn't work either.

corp <- tm_map(corp, content_transformer(stripWhitespace))

Answer

phiver picture phiver · Aug 21, 2018

The code should still be working. You get a warning, not an error. This warning only appears when you have a corpus based on a VectorSource in combination when you use Corpus instead of VCorpus.

The reason is that there is a check in the underlying code to see if the number of names of the corpus content matches the length of the corpus content. With reading the text as a vector there are no document names and this warning pops up. And this is only a warning, no documents have been dropped.

See the difference between the 2 examples

library(tm)

text <- c("this is my text with some other text and some more")

# warning based on Corpus and Vectorsource
text_corpus <- Corpus(VectorSource(text))

# warning appears running following line
tm_map(text_corpus, content_transformer(tolower))
<<SimpleCorpus>>
Metadata:  corpus specific: 1, document level (indexed): 0
Content:  documents: 1
Warning message:
In tm_map.SimpleCorpus(text_corpus, content_transformer(tolower)) :
  transformation drops documents

# Using VCorpus
text_corpus <- VCorpus(VectorSource(text))

# warning doesn't appear
tm_map(text_corpus, content_transformer(tolower))
<<VCorpus>>
Metadata:  corpus specific: 0, document level (indexed): 0
Content:  documents: 1
tm_map(text_corpus, content_transformer(tolower))

Related questions

The difference between bracket [ ] and double bracket [[ ]] for accessing the elements of a list or dataframe

R provides two different methods for accessing the elements of a list or data.frame: [] and [[]]. What is the difference between the two an in which situations should I use one over the other?

Read More
Extract string before "|"

I have a data set wherein a column looks like this: ABC|DEF|GHI, ABCD|EFG|HIJK, ABCDE|FGHI|JKL, DEF|GHIJ|KLM, GHI|JKLM|NO|PQRS, BCDE|FGHI|JKL .... and so on I need to extract the characters that …

Read More
Extract standard errors from glm

I did a glm and I just want to extract the standard errors of each coefficient. I saw on the internet the function se.coef() but it doesn't work, it returns "Error: could not find function "se.coef"".

Read More