Re: st: reliability with -icc- and -estat icc-

[Rebecca Pope <rebecca.a.pope@gmail.com>]

Lenny,
I was just addressing your syntax error, not your underlying data
issues. Why would you expect a ratio to increase when you've made the
numerator 0? If you are getting an ICC close to 0, you should think
about what that is telling you about your data.

If you look at e.g. judges.dta (example for -icc-), you'll see that
the results for the ICC is the same regardless of the method that you
use.

webuse judges
icc rating target judge, mixed
xtmixed rating i.judge || _all: R.target, reml var
nlcom exp(_b[lns1_1_1:_cons])^2/(exp(_b[lnsig_e:_cons])^2+exp(_b[lns1_1_1:_cons])^2)

The two ICCs are nearly equal (to 6 decimal places). Using -xtmixed-
will never give you a negative value though.

An aside: "I'm using the ranks (within an individual) instead of the
actual scores."

If you are using rankings (1-11 presumably) within individual rather
than actual scores it isn't clear to me how rater 4 could be "off the
charts" regardless of actual scores assigned. By converting scores to
rankings, you've wiped out the correlation of scores within rater. You
seem to be interested instead in how e.g. app 1 is rated by all 4
raters (correlation within app). If raters 1, 2, 3 all give it a score
of 1 (their preferred app) & rater 4 gives it a 6, you don't want to
drop that info. That is what you are analyzing.

Regards,
Rebecca



On Tue, Feb 26, 2013 at 10:16 AM, Lenny Lesser <lenny3200@gmail.com> wrote:
> Thanks Rebecca,
> With that code, I get the same problem when I eliminate one rater.
>
> the var(rater) goes to zero, which makes my ICC 0, rather go up to a
> higher number as I expected.
>
>
> ---------- Forwarded message ----------
> From: Rebecca Pope <rebecca.a.pope@gmail.com>
> Date: Tue, Feb 26, 2013 at 7:08 AM
> Subject: Re: st: reliability with -icc- and -estat icc-
> To: statalist@hsphsun2.harvard.edu
>
>
> Lenny,
> I don't think you've got the correct syntax for -xtmixed- if you are
> trying to duplicate ANOVA results, which is the type of analysis that
> -icc- appears to conduct (documentation is still limited, so I won't
> swear to anything).
>
> Use this syntax for -xtmixed-:
> xtmixed rank i.Application || _all: R.Rater, reml var
>
> -estat icc- is not a valid post-estimation command after this
> specification. However, you can just use the definition that ICC =
> Var(Rater)/(Var(Rater)+Var(Residual)).
>
> You might also want to take a look at
> http://www.ats.ucla.edu/stat/stata/faq/xtmixed.htm which will give you
> instructions for using -xtmixed- to conduct ANOVA-type analyses (using
> Stata 10, so you'll need to modify somewhat).
>
> Regards,
> Rebecca
>
>
>
> On Mon, Feb 25, 2013 at 10:56 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
>> I have 4 raters that gave a score of 0-100 on 11 smartphone applications.
>> The data is skewed right, as they all got low scores.  I'm using the
>> ranks (within an individual) instead of the actual scores.  I want to
>> know the correlation in ranking between the different raters.
>>
>> I've tried the two commands:
>>
>> -xtmixed rank Application || Rater: , reml
>> -estat icc
>>
>> (icc=0.19)
>>
>> and
>>
>> -icc rank Rater Application, mixed consistency
>>
>> (icc=0.34)
>>
>> They give me two different answers. Which one is correct?
>>
>>
>> Next, we found out that rater 4 was off the charts, and we want to
>> eliminate her and rerun the analysis. When we do this we get wacky
>> ICCs.  In the first method we get an ICC of 2e-26.  In the 2nd method
>> (-icc), we get -.06.  Eliminating any of the other raters gives us
>> ICCs close to the original ICC.  Why are we getting such a crazy
>> number when we eliminate this 4th rater?
>>
>>
>> I'm guessing this might be instability in the model, but I'm not sure
>> how to get around it.
>>
>> Lenny
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/