Re: st: What does "The effect is significant" after xtprobit mean:examining ALL three effects?

[Maarten Buis <maartenlbuis@gmail.com>]

On Tue, Feb 21, 2012 at 6:29 AM, Amanda Fu wrote:
> I am using an probit random effect estimation (-xtprobit) to examine
> the effect of a key variable (an income shock) on some outcome (work
> or notwork). I want to report if the effect is significant or not.
>
> Previously  statalisters had discussed about this topic. From the
> following link there is a great summary by Mr. Maarten Buis.
> http://www.stata.com/statalist/archive/2008-10/msg01122.html  (Thank
> you, Mr Buis!)
>
> What I am not sure is the last sentence in Mr. Buis's suggestion (see
> below). Does Mr. Buis suggest that when we claim that an effect is
> significant, we usually examine ALL the following three effects and
> ALL of them must reject the null hypotheses?
> (A) the effect on the latent propensity (-probit-),
> (B) the effect on the probability for someone with typical values on
> the explanatory variables
>      (-mfx- or -margeff- with the -at(mean)- option),
> (C) the effect on the probability for a typical person (-margeff-
> without the -at(mean)- option).
>
> What if only two of these effects are significant? My results are that
> only (A) are significant.

Significance refers to a statistical test, and a statistical test
refers to a null hypothesis. A null hypothesis is just one statement,
and the aim of the test is to see whether your data is so inconsistent
with that statement that you can reject the statement. When that
happens we sometimes get lazy and just say that the parameter is
significant. Apparently this short-hand confused you (it is a very
common form of statistical confusion), so the best way is not to be
lazy and spell out the complete sequence, that is, explicitly state
the null-hypothesis, which means you have to choose one of your three
options, state the test statistic and the decision whether or not to
reject the null hypothesis.

So now the question becomes, which of the options A, B, or C do you
choose? If you want "the effect" of a variable, that is, the effect in
the form of one number and not in the form of multiple numbers/a
graph, than you should not use B or C. In essence a marginal effect
tells you how much you expect your probability to increase for a unit
chock in wage. If you report only one such an effect you are in
essence converting your probit model in a linear probability model. If
that is what you want than you should do so directly and check whether
the linearity assumption is reasonable for your data. So if you really
want to report just one additive effect for each variable, than I
would use -xtreg- instead of -xtprobit- with marginal effects. Whether
or not option A is attractive is a matter of taste (your own and your
discipline's). In my and my discipline's taste it is not very
attractive, but tastes notoriously differ. I would also at least
consider -xtlogit- and look at the odds ratios or -xtpoisson- and look
at the risk ratios. For -xtpoisson- you need to check, just like the
linear probability model, whether you get too many implausible
predictions.

The common theme with these recommendations is that you first choose
the kind of effect you want. An effect is just a comparison of groups;
you can look at differences in probabilities between person one unit
wage shock apart, or differences in latent propensity between persons
one unit wage shock apart, or at ratios of probabilities between
persons one unit wage shock apart, or at ratios of odds between
persons one unit wage shock apart. Once you have chosen your effect
you choose the model which directly gives you the effect of your
choice.

There has been a long discussion on this in the thread starting with
<http://www.stata.com/statalist/archive/2012-02/msg00256.html>. In
particular there has been some skepticism about the value of the
linear probability model. Some of that skepticism has been, in my
opinion, too strong, but I agree that there are situations where the
linear probability model is not unproblematic. My only claim is that
reporting one marginal effect per variable is worse than a linear
probability model. So if one has problems with the linear probability
model, than estimating a non-linear model and reporting marginal
effects is certainly not the solution. Instead one should in that
situation rethink the kind of effect one is interested in.

> Additionally, I notice most published papers just say "Marginal
> effects of probit are reported". Which effect do most people mean when
> they simply say "marginal effects"?

That depends to a large extend to when those papers where published
and in what discipline. So to answer that question you really need to
know the state of that (sub-(sub-))discipline at the point that those
articles were written.

Hope this helps,
Maarten

--------------------------
Maarten L. Buis
Institut fuer Soziologie
Universitaet Tuebingen
Wilhelmstrasse 36
72074 Tuebingen
Germany


http://www.maartenbuis.nl
--------------------------

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/