Plotting of very large data sets in R

r plot large-data-volumes
Daniel Arndt picture Daniel Arndt · Dec 3, 2010 · Viewed 16k times · Source

How can I plot a very large data set in R?

I'd like to use a boxplot, or violin plot, or similar. All the data cannot be fit in memory. Can I incrementally read in and calculate the summaries needed to make these plots? If so how?

Answer

mbq picture mbq · Dec 3, 2010

In supplement to my comment to Dmitri answer, a function to calculate quantiles using ff big-data handling package:

ffquantile<-function(ffv,qs=c(0,0.25,0.5,0.75,1),...){
 stopifnot(all(qs<=1 & qs>=0))
 ffsort(ffv,...)->ffvs
 j<-(qs*(length(ffv)-1))+1
 jf<-floor(j);ceiling(j)->jc
 rowSums(matrix(ffvs[c(jf,jc)],length(qs),2))/2
}

This is an exact algorithm, so it uses sorting -- and thus may take a lot of time.

Related questions

Plot two graphs in same plot in R

I would like to plot y1 and y2 in the same plot. x <- seq(-2, 2, 0.05) y1 <- pnorm(x) y2 <- pnorm(x, 1, 1) plot(x, y1, type = "l", col = "red") plot(x, y2, type = "l", col = "…

Read More
How to set limits for axes in ggplot2 R plots?

I plot the following: library(ggplot2) carrots <- data.frame(length = rnorm(500000, 10000, 10000)) cukes <- data.frame(length = rnorm(50000, 10000, 20000)) carrots$veg <- 'carrot' cukes$veg <- 'cuke' vegLengths <- rbind(carrots, cukes) ggplot(vegLengths, aes(length, …

Read More
How to change legend title in ggplot

I have the following plot like below. It was created with this command: library(ggplot2) df <- data.frame(cond = factor(rep(c("A", "B"), each = 200)), rating = c(rnorm(200), rnorm(200, mean=.8))) ggplot(df, aes(x=rating, fill=cond)) + geom_…

Read More