Re: st: Goodness of Fit Measure for Generalized Linear Models with Adjustment for the Number of Parameters

[Roberto Liebscher <roberto.liebscher@ku.de>]

Thanks, Nick. Clearly a R-squared from an OLS model is not comparable 
with a R-squared from a GLM as computed in the before mentioned way. I 
understand your point that for the purpose of comparing non-nested 
models information criteria seem preferable in this case. However, I am 
not a big fan of information criteria because contrary to R-squared they 
do not offer an intuitive understanding. The correlation between 
predicted and actual values adjusted for the number of parameters is 
easier to grasp than minus two times the log likelihood plus two times 
the number of parameters. When I state the adjusted R-squared with the 
number of observations and parameters in the model the reader can easily 
backout the "initial" R-squared. Especially when I fit different 
dependent variables to the same model and report the results in one 
table this procedure is (at least for me) easier to understand and allow 
for the comparison of these models with different endogenous variables. 
But I got your point that this is somewhat a stretch to avoid using AIC 
or BIC.

--
Roberto Liebscher
Catholic University of Eichstaett-Ingolstadt
Department of Business Administration
Chair of Banking and Finance
Auf der Schanz 49
D-85049 Ingolstadt
Germany
Phone:  (+49)-841-937-1929
FAX:    (+49)-841-937-2883
E-mail:   roberto.liebscher@ku-eichstaett.de
Internet: http://www.ku.de/wwf/lfb/


Am 05.03.2014 20:44, schrieb Nick Cox:
> The impulse here is a little puzzling to me. Others here will have a
> deeper mathematical statistics grasp of this than I do, but as I think
> no one has commented I will jump in.
>
> The model you're fitting is estimated using a pseudo- or quasi-maximum
> likelihood procedure. That doesn't rule out calculating an R-squared
> measure as a descriptive or heuristic indicator of goodness of fit,
> which I've been positive about elsewhere e.g.
> http://www.stata.com/support/faq/statistics/r-squared/index.html That
> is a stretch insofar as your fitting is strictly not equivalent to
> maximising R-squared, which is one view of regression. But as long as
> you use words like "heuristic" people may not be too harsh about that.
>
> However, if you now consider the general idea that you should consider
> penalising yourself for using several predictors, the impulse to
> adjust R-squared seems even more of a stretch. If there is a need to
> think about the trade-off between simplicity and fit it is perhaps
> better done using AIC or BIC.
>
> Note that everything is at least a little controversial in this
> territory: most people are moderately fond of some information
> criterion, but there is essentially no agreement that one is best.
> Nick
> njcoxstata@gmail.com
>
>
> On 4 March 2014 17:04, Roberto Liebscher <roberto.liebscher@ku.de> wrote:
> > Hello Statalisters,
> >
> > I model a fractional response variable with a GLM similar to Papke, L.E.,
> > Wooldridge, J.M., 1996. Econometric Methods for Fractional Response
> > Variables with an Application to 401(K) Plan Participation Rates. Journal of
> > Applied Econometrics 11 (1). 619-632.
> >
> > I would like to obtain a goodness-of-fit measure that incorporates the
> > number of parameters in a fashion similar to the adjusted R-squared. It is
> > tempting to compute the correlation between the predicted and the observed
> > values (like in Christopher F Baum's example here:
> > http://fmwww.bc.edu/EC-C/S2013/823/EC823.S2013.nn06.slides.pdf ) and compute
> > the adjusted R-squared according to the formula
> > $R^2-(1-R^2)\frac{p}{n-p-1}$. Since I have never seen something similar in
> > papers so far my question is if there is something wrong about it?
> >
> > Moreover, from a computational point of view one could also estimate the
> > quasi log likelihood function of the unrestricted and the restricted model
> > and follow McFadden's procedure (McFadden's adjusted R^2:
> > http://www.ats.ucla.edu/stat/mult_pkg/faq/general/Psuedo_RSquareds.htm ). If
> > the only goal is to compare non-nested models is there any reason not to use
> > such a measure?
> >
> > Any help is highly appreciated.
> >
> > Roberto
> >
> > --
> > Roberto Liebscher
> > Catholic University of Eichstaett-Ingolstadt
> > Department of Business Administration
> > Chair of Banking and Finance
> > Auf der Schanz 49
> > D-85049 Ingolstadt
> > Germany
> > Phone:  (+49)-841-937-1929
> > FAX:    (+49)-841-937-2883
> > E-mail:   roberto.liebscher@ku-eichstaett.de
> > Internet: http://www.ku.de/wwf/lfb/
> >
> > *
> > *   For searches and help try:
> > *   http://www.stata.com/help.cgi?search
> > *   http://www.stata.com/support/faqs/resources/statalist-faq/
> > *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/