Re: st: Posthoc power analysis for linear mixed effect model

[Mohammod Mostazir <mostazirwho@gmail.com>]

Hi Jeph & Joerg,

Thanks to both of you for your valuable comments and the valuable time
you put into it. Perhaps Stata's 'simpower' does similar thing to what
Jeph suggested and I can see Joerg has valid points too. Actually,
behind the 'posthoc' issue, my intention of this question was to know
about power analysis in mixed effect designs in Stata. Forget about
the posthoc analysis. Say if you were to conduct the same study with
140 cases and you have provisions of 10 repeated measurements, how
would you carryout  the power analysis in Stata given that you know
your future analysis is going to be linear mixed effect designs and
you have the age specific population BMI parameters in hand. One
limitation certainly will be that the population parameters will be
from different groups rather from repeated observations from the same
group. Considering this limitation (trading off with the educated
guess), what will be the Stata procedure for power analysis for such
study.

Thanks.
Mostazir
Research Fellow in Medical Statistics
University of Exeter,
Sir Henry Wellcome Building for Mood Disorders Research
Perry Road, Exeter EX4 4QG
United Kingdom
Phone: +44 (0) 1392 724629
Fax: +44 (0) 1392 724003
web: http://www.exeter.ac.uk/biomedicalhub/team/mrmohammodmostazir/











On 8 March 2014 00:43, Joerg Luedicke <joerg.luedicke@gmail.com> wrote:
>> *  Unless one calculates the curve as you have, one will not know
>>    the power that corresponds to the p-value
>
> But what exactly could one learn from such values? For example, say we
> have a p-value of 0.2 with "observed power" of 0.2, then we could
> _not_ conclude that the test may have yielded an insignificant result
> _because_ of low power. Likewise, some may argue that not only yielded
> their test a significant result, their test was also strongly powered,
> which is a similarly empty argument. Larger p-values always correspond
> to lower "observed power" and the calculation of the latter does not
> add _any_ information.
>
>> *  Most often, one wants to know the power to detect a true effect,
>>    not the observed effect, in which case one cannot infer anything
>>    from the observed effect or the p-value.
>
> I am not sure if I understand this. What often makes sense, however,
> is to simulate data under a variety of assumptions and plausible
> effect sizes, both pro- and retrospectively. For example, it can often
> be very instructive to inspect expected distributions of parameters
> (under certain assumptions and possibly over a range of plausible
> effect sizes) with regard to things like the sign of the effect (e.g.,
> with assumed effect size d under model m, and a given sample size n,
> what would be the probability of an estimated parameter having the
> wrong sign?), it's magnitude etc. which can help to put one's observed
> estimates into perspective. Andrew Gelman & John Carlin call this
> "design calculations" and as they put it: "The relevant question is
> not, "What is the power of a test?" but rather, "What might be
> expected to happen in studies of this size?"" (see:
> http://www.stat.columbia.edu/~gelman/research/unpublished/retropower.pdf)
>
> Joerg
>
> On Fri, Mar 7, 2014 at 5:09 PM, Jeph Herrin <info@flyingbuttress.net> wrote:
>> Yes, but:
>>
>> *  Unless one calculates the curve as you have, one will not know
>>    the power that corresponds to the p-value; and,
>> *  Most often, one wants to know the power to detect a true effect,
>>    not the observed effect, in which case one cannot infer anything
>>    from the observed effect or the p-value.
>>
>> No?
>>
>> Jeph
>>
>>
>>
>> On 3/7/2014 4:43 PM, Joerg Luedicke wrote:
>>>
>>> I'd recommend to not do that at all because a post-hoc power analysis
>>> is a fairly useless endeavor, to say the least. The reason for that is
>>> that the "observed" power, i.e. the calculated power that you obtain
>>> by using the estimates from your model, is a 1:1 function of the
>>> p-values of these estimates. Therefore, calculating post-hoc power
>>> doesn't add any information to what you already have! See Hoenig &
>>> Heisey (2001) for an account on this. Below is an example where we
>>> repeatedly compare means between two groups and store the "observed"
>>> power and p-value from each comparison, then plot power as a function
>>> of p-value:
>>>
>>> * ---------------------------------
>>> cap program drop obsp
>>> program define obsp, rclass
>>>
>>> drop _all
>>> set obs 200
>>> gen x = mod(_n-1,2)
>>> gen e = rnormal()
>>> gen y = 0.1*x + e
>>>
>>> ttest y, by(x)
>>> local p = r(p)
>>> local m1 = r(mu_1)
>>> local m2 = r(mu_2)
>>> local sd1 = r(sd_1)
>>> local sd2 = r(sd_2)
>>>
>>> power twomeans `m1' `m2' , sd1(`sd1') sd2(`sd2') n(200)
>>> return scalar p = `p'
>>> return scalar power = r(power)
>>> end
>>>
>>> simulate power = r(power) p = r(p) , reps(100) seed(1234) : obsp
>>>
>>> scatter power p, connect(l) sort ///
>>> ytitle(`""Observed" power"') ///
>>> xtitle("p-value")
>>> * ---------------------------------
>>>
>>> Joerg
>>>
>>> Reference:
>>> Hoenig, M & DM Heisey (2001): The Abuse of Power: The Pervasive
>>> Fallacy of Power Calculations for Data Analysis. The American
>>> Statistician 55(1): 1-6.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 7, 2014 at 2:55 PM, Mohammod Mostazir <mostazirwho@gmail.com>
>>> wrote:
>>>>
>>>> Dear great stat-warriors,
>>>>
>>>> I need some Stata related H--E--L--P here. I have a dataset that has
>>>> repeated BMI
>>>> (Body Mass Index; continuous scale) measurements of 10 equally spaced
>>>> annual time points from 140 cases. The interest is to observed change
>>>> in BMI in relation to other time-constant and time-varying
>>>> co-variates. The analysis I have carried out is linear mixed effect
>>>> model using Stata's 'xtmixed' command with random intercepts and
>>>> slopes.  Now I would like to carry out a posthoc power analysis to see
>>>> how much power the study has. Is there any light in Stata in relation
>>>> to this? I have seen Stata's ''power repeated'' command which does not
>>>> suit here as they are suitable for one/two way repeated ANOVA designs.
>>>>
>>>> Any comment is highly appreciated. Thanks for reading.
>>>>
>>>> Best,
>>>>
>>>> Mos
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/