Re: st: reliability with -icc- and -estat icc-

[Lenny Lesser <lenny3200@gmail.com>]

Yes. I want to know how consistent the raters are in their scoring
and/or ranking.
The Applications are Fixed Effects.  The raters are Random Effects.

Any help would be appreciated.

I have a colleague who works in SAS and did proc corr alpha.  I'm not
sure if that is the correct way to do it, and I'm not sure that method
is possible in STATA.

On Tue, Feb 26, 2013 at 12:15 PM, JVerkuilen (Gmail)
<jvverkuilen@gmail.com> wrote:
> Just to be clear, you'd like to know the reliability (ICC) of the
> raters using the Applications as fixed effects? So basically you have
> four observations per application, but want to know about how
> consistent the raters are? I'm just trying to make sure I can
> replicate the problem, because when I tried to fit one model I got one
> answer and another blew up.
>
> I ask because I'm writing a paper on ICC estimation and have been
> considering problems very similar to the one you have here.
>
>
> On Tue, Feb 26, 2013 at 2:35 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>> A scatter plot matrix is instructive.
>>
>> Warning: destroys your data.
>>
>> reshape wide Score rank , i(Application) j(Rator)
>> graph matrix Score?
>>
>> #4 really is oddball.
>>
>> Another interesting plot is
>>
>> parplot Score?, tr(raw)
>>
>> where -parplot- must be installed from SSC first.
>>
>> Nick
>>
>> On Tue, Feb 26, 2013 at 7:24 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>> Hi Rebecca,
>>> Thank you for your help.  As a clarification:
>>> We used scores originally, but Rater 4's scores were all very low.
>>> Thus, when we ranked them, there were a lot of ties.  As seen below, 8
>>> of the 11 apps got a rank of "2" by rater.
>>> Lenny
>>>
>>> Application Rator Score rank
>>> 5 1 2 1
>>> 7 1 5 2
>>> 2 1 6 3
>>> 9 1 6 3
>>> 11 1 7 4
>>> 6 1 7 4
>>> 8 1 11 5
>>> 3 1 13 6
>>> 4 1 16 7
>>> 10 1 17 8
>>> 1 1 18 9
>>> 6 2 1 1
>>> 5 2 2 2
>>> 11 2 3 3
>>> 7 2 3 3
>>> 4 2 5 4
>>> 1 2 7 5
>>> 8 2 8 6
>>> 2 2 9 7
>>> 3 2 10 8
>>> 10 2 12 9
>>> 9 2 12 9
>>> 5 3 2 1
>>> 2 3 5 2
>>> 7 3 6 3
>>> 6 3 6 3
>>> 9 3 6 3
>>> 11 3 7 4
>>> 8 3 11 5
>>> 3 3 13 6
>>> 4 3 15 7
>>> 10 3 16 8
>>> 1 3 17 9
>>> 7 4 0 1
>>> 1 4 1 2
>>> 9 4 1 2
>>> 6 4 1 2
>>> 8 4 1 2
>>> 4 4 1 2
>>> 5 4 1 2
>>> 3 4 1 2
>>> 11 4 1 2
>>> 2 4 2 3
>>> 10 4 3 4
>>>
>>> On Tue, Feb 26, 2013 at 9:54 AM, Rebecca Pope <rebecca.a.pope@gmail.com> wrote:
>>>> Lenny,
>>>> I was just addressing your syntax error, not your underlying data
>>>> issues. Why would you expect a ratio to increase when you've made the
>>>> numerator 0? If you are getting an ICC close to 0, you should think
>>>> about what that is telling you about your data.
>>>>
>>>> If you look at e.g. judges.dta (example for -icc-), you'll see that
>>>> the results for the ICC is the same regardless of the method that you
>>>> use.
>>>>
>>>> webuse judges
>>>> icc rating target judge, mixed
>>>> xtmixed rating i.judge || _all: R.target, reml var
>>>> nlcom exp(_b[lns1_1_1:_cons])^2/(exp(_b[lnsig_e:_cons])^2+exp(_b[lns1_1_1:_cons])^2)
>>>>
>>>> The two ICCs are nearly equal (to 6 decimal places). Using -xtmixed-
>>>> will never give you a negative value though.
>>>>
>>>> An aside: "I'm using the ranks (within an individual) instead of the
>>>> actual scores."
>>>>
>>>> If you are using rankings (1-11 presumably) within individual rather
>>>> than actual scores it isn't clear to me how rater 4 could be "off the
>>>> charts" regardless of actual scores assigned. By converting scores to
>>>> rankings, you've wiped out the correlation of scores within rater. You
>>>> seem to be interested instead in how e.g. app 1 is rated by all 4
>>>> raters (correlation within app). If raters 1, 2, 3 all give it a score
>>>> of 1 (their preferred app) & rater 4 gives it a 6, you don't want to
>>>> drop that info. That is what you are analyzing.
>>>>
>>>> Regards,
>>>> Rebecca
>>>>
>>>>
>>>>
>>>> On Tue, Feb 26, 2013 at 10:16 AM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>>>> Thanks Rebecca,
>>>>> With that code, I get the same problem when I eliminate one rater.
>>>>>
>>>>> the var(rater) goes to zero, which makes my ICC 0, rather go up to a
>>>>> higher number as I expected.
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Rebecca Pope <rebecca.a.pope@gmail.com>
>>>>> Date: Tue, Feb 26, 2013 at 7:08 AM
>>>>> Subject: Re: st: reliability with -icc- and -estat icc-
>>>>> To: statalist@hsphsun2.harvard.edu
>>>>>
>>>>>
>>>>> Lenny,
>>>>> I don't think you've got the correct syntax for -xtmixed- if you are
>>>>> trying to duplicate ANOVA results, which is the type of analysis that
>>>>> -icc- appears to conduct (documentation is still limited, so I won't
>>>>> swear to anything).
>>>>>
>>>>> Use this syntax for -xtmixed-:
>>>>> xtmixed rank i.Application || _all: R.Rater, reml var
>>>>>
>>>>> -estat icc- is not a valid post-estimation command after this
>>>>> specification. However, you can just use the definition that ICC =
>>>>> Var(Rater)/(Var(Rater)+Var(Residual)).
>>>>>
>>>>> You might also want to take a look at
>>>>> http://www.ats.ucla.edu/stat/stata/faq/xtmixed.htm which will give you
>>>>> instructions for using -xtmixed- to conduct ANOVA-type analyses (using
>>>>> Stata 10, so you'll need to modify somewhat).
>>>>>
>>>>> Regards,
>>>>> Rebecca
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 25, 2013 at 10:56 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>>>>> I have 4 raters that gave a score of 0-100 on 11 smartphone applications.
>>>>>> The data is skewed right, as they all got low scores.  I'm using the
>>>>>> ranks (within an individual) instead of the actual scores.  I want to
>>>>>> know the correlation in ranking between the different raters.
>>>>>>
>>>>>> I've tried the two commands:
>>>>>>
>>>>>> -xtmixed rank Application || Rater: , reml
>>>>>> -estat icc
>>>>>>
>>>>>> (icc=0.19)
>>>>>>
>>>>>> and
>>>>>>
>>>>>> -icc rank Rater Application, mixed consistency
>>>>>>
>>>>>> (icc=0.34)
>>>>>>
>>>>>> They give me two different answers. Which one is correct?
>>>>>>
>>>>>>
>>>>>> Next, we found out that rater 4 was off the charts, and we want to
>>>>>> eliminate her and rerun the analysis. When we do this we get wacky
>>>>>> ICCs.  In the first method we get an ICC of 2e-26.  In the 2nd method
>>>>>> (-icc), we get -.06.  Eliminating any of the other raters gives us
>>>>>> ICCs close to the original ICC.  Why are we getting such a crazy
>>>>>> number when we eliminate this 4th rater?
>>>>>>
>>>>>>
>>>>>> I'm guessing this might be instability in the model, but I'm not sure
>>>>>> how to get around it.
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>
>
>
> --
> JVVerkuilen, PhD
> jvverkuilen@gmail.com
>
> "It is like a finger pointing away to the moon. Do not concentrate on
> the finger or you will miss all that heavenly glory." --Bruce Lee,
> Enter the Dragon (1973)
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/