Re: st: odd simulation results for ICC with binary outcomes

[Rebecca Pope <rebecca.a.pope@gmail.com>]

Thanks Austin. With 1,000 reps, I would expect the same general shape
by the CLT, correct? What I would expect to differ would be the mean
around which they converge. With 1,000 reps, we can't reject equality
between the simulated beta-bin & 1/11; with 10,000 we can. The MM
estimator similarly "flip-flops", but the ANOVA doesn't. Just looking
at the graphs, I agree that it is difficult to argue that the plotted
distributions are different (regardless of # of reps chosen); the
lines are nearly indistinguishable from each other. Indicating, I
think, that despite the tests they are probably heading in the same
direction.

I've done some more experimenting with beta distributions that may be
more realistic. The literature indicates that often individuals do not
get recommended care, so I tried some distributions that put more
weight on the lower probabilities of success. It takes 10,000
repetitions to achieve the normal shape from the sample means with the
skewed distributions. Once this happens, all of the modes (and hence
means) look pretty close to the true ICC. The t-tests however indicate
that they haven't converged to the theoretically true value, except
for the ANOVA.

Finally, the beta-bin does even worse with unequally sized groups. The
ANOVA still does just fine. I might be focusing over-much on the
tests, but this seems completely counterintuitive to me. Am I just
misunderstanding how the simulations should work? Or is it time to
just accept that regardless of intuition, the ANOVA is going to do
just fine irrespective of the underlying distribution? The Donner
article seemed to suggest that this is the case, at least for some
non-normal distributions. There are also several references at the end
regarding binomial data that I will try to find.

Thanks also for cleaning up some of the finer points of my code. I
haven't used -simulate- often, but of course -tempvar- is unnecessary
in that context. I didn't think about that. It also (obviously) didn't
occur to me to use -scalar- in place of -local-.

Regards,
Rebecca

On Fri, Feb 15, 2013 at 2:43 PM, Austin Nichols <austinnichols@gmail.com> wrote:
> Rebecca Pope <rebecca.a.pope@gmail.com>:
> I don't see much difference across estimators below.
> You might also want to read
>  http://www.jstor.org/stable/1403259
> for more options.
>
> set seed 72114
> set more off
> clear all
> program define betabinsim, rclass
>  version 12
>  syntax [, a(real 5) b(real 5) groups(int 135) n(real 100)]
>  drop _all
>  set obs `groups'
>  gen int group = _n
>  gen y1 = rbinomial(`n', rbeta(`a',`b'))
>  gen y0 = `n'-y1
>  reshape long y, i(group) j(event)
>  xtnbreg y event, i(group) fe nolog
>  scalar alph=exp(_b[_cons]+_b[event])
>  scalar beta=exp(_b[_cons])
>  return scalar n = `n'
>  return scalar a = alph
>  return scalar b = beta
>  return scalar mu = alph/(alph+beta)
>  return scalar icc = 1/(alph+beta+1)
>  expandcl y, cluster(group event) generate(foo)
>  loneway event group
>  return scalar icc_anova = r(rho)
>  egen double mn=mean(event), by(group)
>  gen double se2=mn*(1-mn)/`n'
>  su se2, mean
>  scalar corrct=r(mean)
>  su mn
>  scalar Var=r(Var)-corrct
>  ret scalar iccm=Var/(r(mean)*(1-r(mean)))
>  eret clear
> end
> simul, r(1000): betabinsim
> local truerho = 1/(1+5+5)
> ttest icc==icc_anova, unpaired
> ttest icc ==`truerho'
> ttest icc_anova == `truerho'
> ttest iccm == `truerho'
> la var icc_an "ICC from ANOVA"
> la var icc "ICC from xtnbreg"
> la var iccm "ICC from MM"
> tw kdensity icc_an||kdensity iccm||kdensity icc, xli(`=1/11')
> su
>
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/