Re: st: reliability with -icc- and -estat icc-

[Lenny Lesser <lenny3200@gmail.com>]

Hi Rebecca,
Thank you for your help.  As a clarification:
We used scores originally, but Rater 4's scores were all very low.
Thus, when we ranked them, there were a lot of ties.  As seen below, 8
of the 11 apps got a rank of "2" by rater.
Lenny

Application Rator Score rank
5 1 2 1
7 1 5 2
2 1 6 3
9 1 6 3
11 1 7 4
6 1 7 4
8 1 11 5
3 1 13 6
4 1 16 7
10 1 17 8
1 1 18 9
6 2 1 1
5 2 2 2
11 2 3 3
7 2 3 3
4 2 5 4
1 2 7 5
8 2 8 6
2 2 9 7
3 2 10 8
10 2 12 9
9 2 12 9
5 3 2 1
2 3 5 2
7 3 6 3
6 3 6 3
9 3 6 3
11 3 7 4
8 3 11 5
3 3 13 6
4 3 15 7
10 3 16 8
1 3 17 9
7 4 0 1
1 4 1 2
9 4 1 2
6 4 1 2
8 4 1 2
4 4 1 2
5 4 1 2
3 4 1 2
11 4 1 2
2 4 2 3
10 4 3 4

On Tue, Feb 26, 2013 at 9:54 AM, Rebecca Pope <rebecca.a.pope@gmail.com> wrote:
> Lenny,
> I was just addressing your syntax error, not your underlying data
> issues. Why would you expect a ratio to increase when you've made the
> numerator 0? If you are getting an ICC close to 0, you should think
> about what that is telling you about your data.
>
> If you look at e.g. judges.dta (example for -icc-), you'll see that
> the results for the ICC is the same regardless of the method that you
> use.
>
> webuse judges
> icc rating target judge, mixed
> xtmixed rating i.judge || _all: R.target, reml var
> nlcom exp(_b[lns1_1_1:_cons])^2/(exp(_b[lnsig_e:_cons])^2+exp(_b[lns1_1_1:_cons])^2)
>
> The two ICCs are nearly equal (to 6 decimal places). Using -xtmixed-
> will never give you a negative value though.
>
> An aside: "I'm using the ranks (within an individual) instead of the
> actual scores."
>
> If you are using rankings (1-11 presumably) within individual rather
> than actual scores it isn't clear to me how rater 4 could be "off the
> charts" regardless of actual scores assigned. By converting scores to
> rankings, you've wiped out the correlation of scores within rater. You
> seem to be interested instead in how e.g. app 1 is rated by all 4
> raters (correlation within app). If raters 1, 2, 3 all give it a score
> of 1 (their preferred app) & rater 4 gives it a 6, you don't want to
> drop that info. That is what you are analyzing.
>
> Regards,
> Rebecca
>
>
>
> On Tue, Feb 26, 2013 at 10:16 AM, Lenny Lesser <lenny3200@gmail.com> wrote:
>> Thanks Rebecca,
>> With that code, I get the same problem when I eliminate one rater.
>>
>> the var(rater) goes to zero, which makes my ICC 0, rather go up to a
>> higher number as I expected.
>>
>>
>> ---------- Forwarded message ----------
>> From: Rebecca Pope <rebecca.a.pope@gmail.com>
>> Date: Tue, Feb 26, 2013 at 7:08 AM
>> Subject: Re: st: reliability with -icc- and -estat icc-
>> To: statalist@hsphsun2.harvard.edu
>>
>>
>> Lenny,
>> I don't think you've got the correct syntax for -xtmixed- if you are
>> trying to duplicate ANOVA results, which is the type of analysis that
>> -icc- appears to conduct (documentation is still limited, so I won't
>> swear to anything).
>>
>> Use this syntax for -xtmixed-:
>> xtmixed rank i.Application || _all: R.Rater, reml var
>>
>> -estat icc- is not a valid post-estimation command after this
>> specification. However, you can just use the definition that ICC =
>> Var(Rater)/(Var(Rater)+Var(Residual)).
>>
>> You might also want to take a look at
>> http://www.ats.ucla.edu/stat/stata/faq/xtmixed.htm which will give you
>> instructions for using -xtmixed- to conduct ANOVA-type analyses (using
>> Stata 10, so you'll need to modify somewhat).
>>
>> Regards,
>> Rebecca
>>
>>
>>
>> On Mon, Feb 25, 2013 at 10:56 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>> I have 4 raters that gave a score of 0-100 on 11 smartphone applications.
>>> The data is skewed right, as they all got low scores.  I'm using the
>>> ranks (within an individual) instead of the actual scores.  I want to
>>> know the correlation in ranking between the different raters.
>>>
>>> I've tried the two commands:
>>>
>>> -xtmixed rank Application || Rater: , reml
>>> -estat icc
>>>
>>> (icc=0.19)
>>>
>>> and
>>>
>>> -icc rank Rater Application, mixed consistency
>>>
>>> (icc=0.34)
>>>
>>> They give me two different answers. Which one is correct?
>>>
>>>
>>> Next, we found out that rater 4 was off the charts, and we want to
>>> eliminate her and rerun the analysis. When we do this we get wacky
>>> ICCs.  In the first method we get an ICC of 2e-26.  In the 2nd method
>>> (-icc), we get -.06.  Eliminating any of the other raters gives us
>>> ICCs close to the original ICC.  Why are we getting such a crazy
>>> number when we eliminate this 4th rater?
>>>
>>>
>>> I'm guessing this might be instability in the model, but I'm not sure
>>> how to get around it.
>>>
>>> Lenny
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/