Re: st: criticisms of classical model selection methods

[Maarten buis <maartenbuis@yahoo.co.uk>]

--- On Thu, 19/8/10, Sam Brilleman wrote:
> Is anyone able to point me in the direction of references
> dealing with the discussion for and against traditional
> methods of model selection. I am particularly interested in
> criticisms of the assumptions on which commonly used model
> selection criteria are based (eg. AIC, BIC, etc).

Raftery, Adrian E. 1995. "Bayesian Model Selection in Social Research." Sociological Methodology 25:111–163.
And in particular the comments that follow this article

G. Claeskens and Nils Lid Hjort (2008) Model Selection and Model 
Averaging, Cambridge University Press

Burnham, K. P., and D. R. Anderson.  2002.  Model selection and 
multimodel inference: a practical information-theoretic approach. 
2nd Edition. Springer-Verlag, New York, New York.
 
> I'm also quite interested to hear what measures people
> often use when selecting between competing non-nested
> models, particularly when the models are non-linear. Is AIC
> reasonable when the models differ only by one covariate, or
> is it always worthwhile making judgement on a number of GOF
> tests?

This very much depends on what the aim of the model selection is:

If the comparison of models is done to test a specific hypothesis
then it is nice to stick to a single logic on which to base your
decision to choose one model over the other. All these logics are
flawed, so the best thing you can do is choose the one that is
most commonly used in your (sub-sub-)discipline.

If the comparison of models is done to find the "best" model, 
then you need to start with defining what "best" means. Almost
always you find that there is no way you can cover that with
one goodness of fit statistic, or even many of these statistics.

Hope this helps,
Maarten

--------------------------
Maarten L. Buis
Institut fuer Soziologie
Universitaet Tuebingen
Wilhelmstrasse 36
72074 Tuebingen
Germany

http://www.maartenbuis.nl
--------------------------


      

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/