I am trying to run a mixed-effects model that predicts F2_difference
with the rest of the columns as predictors, but I get an error message that says
fixed-effect model matrix is rank deficient so dropping 7 columns / coefficients.
From this link, Fixed-effects model is rank deficient, I think I should use findLinearCombos
in the R package caret
. However, when I try findLinearCombos(data.df)
, it gives me the error message
Error in qr.default(object) : NA/NaN/Inf in foreign function call (arg 1) In addition: Warning message: In qr.default(object) : NAs introduced by coercion
My data does not have any NAs - What could be causing this? (Sorry if the answer is various obvious - I am new to R).
All of my data are factors except the numerical value that I am trying to predict. Here is a small sample of my data.
sex <- c("f", "m", "f", "m")
nasal <- c("TRUE", "TRUE", "FALSE", "FALSE")
vowelLabel <- c("a", "e", "i", "o")
speaker <- c("Jim", "John", "Ben", "Sally")
word_1 <- c("going", "back", "bag", "back")
type <- c("coronal", "coronal", "labial", "velar")
F2_difference <- c(345.6, -765.8, 800, 900.5)
data.df <- data.frame(sex, nasal, vowelLabel, speaker,
word_1, type, F2_difference
stringsAsFactors = TRUE)
Edit: Here is some more code, if it helps.
formula <- F2_difference ~ sex + nasal + type + vowelLabel +
type * vowelLabel + nasal * type +
(1|speaker) + (1|word_1)
lmer(formula, REML = FALSE, data = data.df)
Editor edit:
The OP did not provide sufficient number of test data to allow an actual run of the model in lmer
for the reader. But this is not too big a issue. This is still a very good post!
You are slightly over-concerned with the warning message:
fixed-effect model matrix is rank deficient so dropping 7 columns / coefficients.
It is a warning not an error. There is neither misuse of lmer
nor ill-specification of model formula, thus you will obtain an estimated model. But to answer your question, I shall strive to explain it.
During execution of lmer
, your model formula is broken into a fixed effect formula and a random effect formula, and for each a model matrix is constructed. Construction for the fixed one is via the standard model matrix constructor model.matrix
; construction for the random one is complicated but not related to your question, so I just skip it.
For your model, you can check what the fixed effect model matrix looks like by:
fix.formula <- F2_difference ~ sex + nasal + type + vowelLabel +
type * vowelLabel + nasal * type
X <- model.matrix (fix.formula, data.df)
All your variables are factor so X
will be binary. Though model.matrix
applies contrasts
for each factor and their interaction, it is still possible that X
does not end up with full column rank, as a column may be a linear combination of some others (which can either be precise or numerically close). In your case, some levels of one factor may be nested in some levels of another.
Rank deficiency can arise in many different ways. The other answer shares a CrossValidated answer offering substantial discussions, on which I will make some comments.
So, sometimes we can workaround the deficiency but it is not always possible to achieve this. Thus, any well-written model fitting routine, like lm
, glm
, mgcv::gam
, will apply QR decomposition for X
to only use its full-rank subspace, i.e., a maximum subset of X
's columns that gives a full-rank space, for estimation, fixing coefficients associated with the rest of the columns at 0 or NA
. The warning you got just implies this. There are originally ncol(X)
coefficients to estimate, but due to deficiency, only ncol(X) - 7
will be estimated, with the rest being 0 or NA
. Such numerical workaround ensures that a least squares solution can be obtained in the most stable manner.
To better digest this issue, you can use lm
to fit a linear model with fix.formula
.
fix.fit <- lm(fix.formula, data.df, method = "qr", singular.ok = TRUE)
method = "qr"
and singular.ok = TRUE
are default, so actually we don't need to set it. But if we specify singular.ok = FALSE
, lm
will stop and complain about rank-deficiency.
lm(fix.formula, data.df, method = "qr", singular.ok = FALSE)
#Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) :
# singular fit encountered
You can then check the returned values in fix.fit
.
p <- length(coef)
coef <- fix.fit$coef
no.NA <- sum(is.na(coef))
rank <- fix.fit$rank
It is guaranteed that p = ncol(X)
, but you should see no.NA = 7
and rank + no.NA = p
.
Exactly the same thing happens inside lmer
. lm
will not report deficiency while lmer
does. This is in fact informative, as too often, I see people asking why lm
returns NA
for some coefficients.
Update 1 (2016-05-07):
Let me see if I have this right: The short version is that one of my predictor variables is correlated with another, but I shouldn't worry about it. It is appropriate to use factors, correct? And I can still compare models with
anova
or by looking at the BIC?
Don't worry about the use of summary
or anova
. Methods are written so that the correct number of parameters (degree of freedom) will be used to produce valid summary statistics.
Update 2 (2016-11-06):
Let's also hear what package author of lme4
would say: rank deficiency warning mixed model lmer. Ben Bolker has mentioned caret::findLinearCombos
, too, particularly because the OP there want to address deficiency issue himself.
Update 3 (2018-07-27):
Rank-deficiency is not a problem for valid model estimation and comparison, but could be a hazard in prediction. I recently composed a detailed answer with simulated examples on CrossValidated: R lm
, Could anyone give me an example of the misleading case on “prediction from a rank-deficient”? So, yes, in theory we should avoid rank-deficient estimation. But in reality, there is no so-called "true model": we try to learn it from data. We can never compare an estimated model to "truth"; the best bet is to choose the best one from a number of models we've built. So if the "best" model ends up rank-deficient, we can be skeptical about it but probably there is nothing we can do immediately.