Re: st: Warning: variance matrix is nonsymmetric or highly singular

[jpitblado@stata.com (Jeff Pitblado, StataCorp LP)]

Melissa A. Clark <maclark@alumni.princeton.edu> is using Stata's -svy-
estimation commands with sparse predictors and is getting a warning about the
variance matrix resulting in coefficient table with all missing standard
errors:

> A while back, someone wrote to ask about the message "Warning:
> variance matrix is nonsymmetric or highly singular," which appears in
> Stata 9 and 10 survey commands but not in earlier versions. When this
> message appears, the regression results display coefficient estimates
> but no standard errors for any of the coefficients.
> 
> Jeff Pitblado posted the response that this message "is most likely
> due to one or more sparse indicator variables.  By sparse indicator, I
> mean a variable that takes on the values 0 or 1 (or missing) and is 1
> for a very small proportion of the observations. The best example is
> an indicator variable that identifies 1 observation; this will
> invalidate the large sample theory that the robust variance estimator
> depends upon for the coefficient on this variable -- however, you can
> simply remove this variable from the model to solve the problem."
> 
> I understand why Stata cannot estimate the standard error for the
> "problem" covariate.  However, earlier versions of Stata display a
> missing standard error for the problem covariate, but display
> nonmissing SEs for the other covariates.

This is not exactly accurate.  We changed -ereturn post- so that it would
display a warning message (and missing standard errors) under certain
circumstances that previously would result in Stata stopping with an error
message.

Thus performing the same analysis using an earlier Stata would result in an
error message with no results displayed.

> My understanding is that these other SE estimates are fine (and would
> presumably be the same if we removed the problem covariate from the model
> and estimated in Stata 9).
>
> I would ideally like to be able to run the models in Stata 9 without
> having to drop the problem covariates, and to have Stata display SEs
> for the non-problem coefficients.  Is there any way to do this? And am
> I correct that the estimated SEs on the non-problem coefficients (eg
> if estimated in Stata 8) should in fact be correct?

We sympathise with Melissa's dilemma, but the problem is that we are not clear
on what the criterion for dropping sparse indicators should be.

The data analyst is in the best position to judge which of the sparse
indicators can/should be dropped.

--Jeff
jpitblado@stata.com
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/