Re: st: Posthoc power analysis for linear mixed effect model

[Joerg Luedicke <joerg.luedicke@gmail.com>]

> *  Unless one calculates the curve as you have, one will not know
>    the power that corresponds to the p-value

But what exactly could one learn from such values? For example, say we
have a p-value of 0.2 with "observed power" of 0.2, then we could
_not_ conclude that the test may have yielded an insignificant result
_because_ of low power. Likewise, some may argue that not only yielded
their test a significant result, their test was also strongly powered,
which is a similarly empty argument. Larger p-values always correspond
to lower "observed power" and the calculation of the latter does not
add _any_ information.

> *  Most often, one wants to know the power to detect a true effect,
>    not the observed effect, in which case one cannot infer anything
>    from the observed effect or the p-value.

I am not sure if I understand this. What often makes sense, however,
is to simulate data under a variety of assumptions and plausible
effect sizes, both pro- and retrospectively. For example, it can often
be very instructive to inspect expected distributions of parameters
(under certain assumptions and possibly over a range of plausible
effect sizes) with regard to things like the sign of the effect (e.g.,
with assumed effect size d under model m, and a given sample size n,
what would be the probability of an estimated parameter having the
wrong sign?), it's magnitude etc. which can help to put one's observed
estimates into perspective. Andrew Gelman & John Carlin call this
"design calculations" and as they put it: "The relevant question is
not, "What is the power of a test?" but rather, "What might be
expected to happen in studies of this size?"" (see:
http://www.stat.columbia.edu/~gelman/research/unpublished/retropower.pdf)

Joerg

On Fri, Mar 7, 2014 at 5:09 PM, Jeph Herrin <info@flyingbuttress.net> wrote:
> Yes, but:
>
> *  Unless one calculates the curve as you have, one will not know
>    the power that corresponds to the p-value; and,
> *  Most often, one wants to know the power to detect a true effect,
>    not the observed effect, in which case one cannot infer anything
>    from the observed effect or the p-value.
>
> No?
>
> Jeph
>
>
>
> On 3/7/2014 4:43 PM, Joerg Luedicke wrote:
>>
>> I'd recommend to not do that at all because a post-hoc power analysis
>> is a fairly useless endeavor, to say the least. The reason for that is
>> that the "observed" power, i.e. the calculated power that you obtain
>> by using the estimates from your model, is a 1:1 function of the
>> p-values of these estimates. Therefore, calculating post-hoc power
>> doesn't add any information to what you already have! See Hoenig &
>> Heisey (2001) for an account on this. Below is an example where we
>> repeatedly compare means between two groups and store the "observed"
>> power and p-value from each comparison, then plot power as a function
>> of p-value:
>>
>> * ---------------------------------
>> cap program drop obsp
>> program define obsp, rclass
>>
>> drop _all
>> set obs 200
>> gen x = mod(_n-1,2)
>> gen e = rnormal()
>> gen y = 0.1*x + e
>>
>> ttest y, by(x)
>> local p = r(p)
>> local m1 = r(mu_1)
>> local m2 = r(mu_2)
>> local sd1 = r(sd_1)
>> local sd2 = r(sd_2)
>>
>> power twomeans `m1' `m2' , sd1(`sd1') sd2(`sd2') n(200)
>> return scalar p = `p'
>> return scalar power = r(power)
>> end
>>
>> simulate power = r(power) p = r(p) , reps(100) seed(1234) : obsp
>>
>> scatter power p, connect(l) sort ///
>> ytitle(`""Observed" power"') ///
>> xtitle("p-value")
>> * ---------------------------------
>>
>> Joerg
>>
>> Reference:
>> Hoenig, M & DM Heisey (2001): The Abuse of Power: The Pervasive
>> Fallacy of Power Calculations for Data Analysis. The American
>> Statistician 55(1): 1-6.
>>
>>
>>
>>
>> On Fri, Mar 7, 2014 at 2:55 PM, Mohammod Mostazir <mostazirwho@gmail.com>
>> wrote:
>>>
>>> Dear great stat-warriors,
>>>
>>> I need some Stata related H--E--L--P here. I have a dataset that has
>>> repeated BMI
>>> (Body Mass Index; continuous scale) measurements of 10 equally spaced
>>> annual time points from 140 cases. The interest is to observed change
>>> in BMI in relation to other time-constant and time-varying
>>> co-variates. The analysis I have carried out is linear mixed effect
>>> model using Stata's 'xtmixed' command with random intercepts and
>>> slopes.  Now I would like to carry out a posthoc power analysis to see
>>> how much power the study has. Is there any light in Stata in relation
>>> to this? I have seen Stata's ''power repeated'' command which does not
>>> suit here as they are suitable for one/two way repeated ANOVA designs.
>>>
>>> Any comment is highly appreciated. Thanks for reading.
>>>
>>> Best,
>>>
>>> Mos
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/