Remove rows where all variables are NA using dplyr

r dplyr tidyverse
hejseb picture hejseb · Jan 12, 2017 · Viewed 19.1k times · Source

I'm having some issues with a seemingly simple task: to remove all rows where all variables are NA using dplyr. I know it can be done using base R (Remove rows in R matrix where all data is NA and Removing empty rows of a data file in R), but I'm curious to know if there is a simple way of doing it using dplyr.

Example:

library(tidyverse)
dat <- tibble(a = c(1, 2, NA), b = c(1, NA, NA), c = c(2, NA, NA))
filter(dat, !is.na(a) | !is.na(b) | !is.na(c))

The filter call above does what I want but it's infeasible in the situation I'm facing (as there is a large number of variables). I guess one could do it by using filter_ and first creating a string with the (long) logical statement, but it seems like there should be a simpler way.

Another way is to use rowwise() and do():

na <- dat %>% 
  rowwise() %>% 
  do(tibble(na = !all(is.na(.)))) %>% 
  .$na
filter(dat, na)

but that does not look too nice, although it gets the job done. Other ideas?

Answer

MarkusN picture MarkusN · May 2, 2018

Since dplyr 0.7.0 new, scoped filtering verbs exists. Using filter_any you can easily filter rows with at least one non-missing column:

dat %>% filter_all(any_vars(!is.na(.)))

Using @hejseb benchmarking algorithm it appears that this solution is as efficient as f4.

Related questions

Removing NA observations with dplyr::filter()

My data looks like this: library(tidyverse) df <- tribble( ~a, ~b, ~c, 1, 2, 3, 1, NA, 3, NA, 2, 3 ) I can remove all NA observations with drop_na(): df %>% drop_na() Or remove all NA observations in a single column (a for …

Read More
Error in bind_rows_(x, .id) : Argument 1 must have names

Here is a code snippet: y <- purrr::map(1:2, ~ c(a=.x)) test1 <- dplyr::bind_rows(y) test2 <- do.call(dplyr::bind_rows, y) The first call to bind_rows (test1) generates the error Error …

Read More
How to reorder factor levels in a tidy way?

Hi I usually use some code like the following to reorder bars in ggplot or other types of plots. Normal plot (unordered) library(tidyverse) iris.tr <-iris %>% group_by(Species) %>% mutate(mSW = mean(Sepal.Width)) %>% select(…

Read More