Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: reliability with -icc- and -estat icc-
From
"JVerkuilen (Gmail)" <[email protected]>
To
[email protected]
Subject
Re: st: reliability with -icc- and -estat icc-
Date
Tue, 26 Feb 2013 20:40:49 -0500
On Tue, Feb 26, 2013 at 8:31 PM, Lenny Lesser <[email protected]> wrote:
> Yes. I want to know how consistent the raters are in their scoring
> and/or ranking.
> The Applications are Fixed Effects. The raters are Random Effects.
>
> Any help would be appreciated.
>
> I have a colleague who works in SAS and did proc corr alpha. I'm not
> sure if that is the correct way to do it, and I'm not sure that method
> is possible in STATA.
It's absolutely possible. I just ran the following model, which I
believe (but am not 100% sure) is what you want. This has a random
intercept for Rator and fixed effects for application. The ICC is
massively inflated by rater 4, who is clearly anchored very
differently and has a massively lower response variance. HUGE outlier.
If you'd be willing I'd love to use it as an example for my Bayesian
ICC estimator paper.
. xtmixed Score i.Application if Rator != 4, || Rator:,
covariance(independent) difficult
Mixed-effects ML regression Number of obs = 33
Group variable: Rator Number of groups = 3
Obs per group: min = 11
avg = 11.0
max = 11
Wald chi2(10) = 95.51
Log likelihood = -77.750467 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
Score | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
Application |
2 | -7.333333 1.954361 -3.75 0.000 -11.16381 -3.502855
3 | -2 1.954361 -1.02 0.306 -5.830478 1.830478
4 | -2 1.954361 -1.02 0.306 -5.830478 1.830478
5 | -12 1.954361 -6.14 0.000 -15.83048 -8.169522
6 | -9.333333 1.954361 -4.78 0.000 -13.16381 -5.502855
7 | -9.333333 1.954361 -4.78 0.000 -13.16381 -5.502855
8 | -4 1.954361 -2.05 0.041 -7.830478 -.1695221
9 | -6 1.954361 -3.07 0.002 -9.830478 -2.169522
10 | 1 1.954361 0.51 0.609 -2.830478 4.830478
11 | -8.333333 1.954361 -4.26 0.000 -12.16381 -4.502855
|
_cons | 14 1.565624 8.94 0.000 10.93143 17.06857
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
Rator: Identity |
sd(_cons) | 1.274458 .6891609 .4416111 3.677996
-----------------------------+------------------------------------------------
sd(Residual) | 2.393594 .3090117 1.858493 3.082763
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) = 3.99 Prob >= chibar2 = 0.0229
. estat icc
Residual intraclass correlation
------------------------------------------------------------------------------
Level | ICC Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
Rator | .2208793 .1946277 .0299686 .7223361
------------------------------------------------------------------------------
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/