RE: st: Hosmer-Lemeshow and other Pseudo Rsquares

["Lachenbruch, Peter" <Peter.Lachenbruch@oregonstate.edu>]

I just had reason to check Frank Harrell's book (REgression MOdeling Strategies, Springer 2001) on variable selection and his advice was fit the full model with variables taht are defined by their contextual (biological, sociological) importance.  Don't believe any R^2 or other values as they are distorted by the selection process. 

More recently, the LASSO (Tibshirani - don't have reference, but google scholar can provide one) or LARS (Efron, Hastie) should be helpful.

Tony

________________________________________
From: owner-statalist@hsphsun2.harvard.edu [owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox [njcoxstata@gmail.com]
Sent: Monday, May 14, 2012 7:10 AM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: Hosmer-Lemeshow and other Pseudo Rsquares

I suggest a few meta-rules for yourself:

1. Whatever you calculate should be defined and calculated
consistently across different models.

2. Whatever you calculate you promise to use with extreme caution
always flagging precisely how it is calculated.

3. You don't decide which model is "best" from these measures; you
just treat them as descriptive statistics.

#1 sounds easy but can bite quite hard. I find the idea of R^2 as

square of correlation between observed and predicted

as the sense of R^2 that I like best but this grows out of a long
personal history of working with correlation and regression and one
that is dominated by working with continuous outcomes. People with a
long history the other way round might want you to look for

1 - (log likelihood for model) / (log likelihood for same model with
only a constant term)

and could have similar warm feelings for that. Others would find the
whole idea of looking at goodness of fit without also assessing number
of parameters or model complexity in general quite misguided, but
those others can't agree on which of various *IC you should use, and
even those who have a favourite often say, "You should use ?IC except
that it usually favours over-simplified models" or some such.

On the first option see

FAQ     . . . . . . . . . . . . . . . . . . . . . . . Do-it-yourself R-squared
        . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  N. J. Cox
        9/03    How can I get an R-squared value when a Stata command
                does not supply one?
                http://www.stata.com/support/faqs/stat/rsquared.html

On Mon, May 14, 2012 at 2:47 PM, Joseph Padgett <josephpadgett@gmail.com> wrote:

> I am working with a data set where students are nested within school.
>
> I have completed a thorough run of models starting with nulls and
> ending with full fixed- and random-effects with all controls and
> predictors and several models in between with various combinations of
> controls.  My dependent variable is a binary outcome.
>
> I have Haussman tests, LR tests, and Wald taken care of, but I would
> like to report some goodness-of-fit results for my models.  I am aware
> of the Hosmer-Lemeshow test statistic and it's interpretation, but I'm
> having a difficult time finding out how to compute it from my model
> results.  I would also like to consider alternatives such as Cox and
> Snell.
>
> I have run my models with each of xtlogit, xtmelogit, and gllamm.  I
> did this mostly to be able to learn a bit about the post estimation
> commands and different options with each command.  That being said, I
> don't know how to get the pseudo Rsquare measures after any of these
> and most explanations that I find refer only to the logit command and
> give examples using very simplistic models.
>
> I'm fairly certain there's something terribly obvious that I'm
> overlooking.  Any help would be greatly appreciated.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/