Re: st: reliability with -icc- and -estat icc-

[Nick Cox <njcoxstata@gmail.com>]

There is an -alpha- command in Stata [NB] and -search alpha- would
have pointed you to it.

It strikes me that there is little or no point in any calculation that
includes rater 4.

Nick

On Wed, Feb 27, 2013 at 1:31 AM, Lenny Lesser <lenny3200@gmail.com> wrote:
> Yes. I want to know how consistent the raters are in their scoring
> and/or ranking.
> The Applications are Fixed Effects.  The raters are Random Effects.
>
> Any help would be appreciated.
>
> I have a colleague who works in SAS and did proc corr alpha.  I'm not
> sure if that is the correct way to do it, and I'm not sure that method
> is possible in STATA.
>
> On Tue, Feb 26, 2013 at 12:15 PM, JVerkuilen (Gmail)
> <jvverkuilen@gmail.com> wrote:
>> Just to be clear, you'd like to know the reliability (ICC) of the
>> raters using the Applications as fixed effects? So basically you have
>> four observations per application, but want to know about how
>> consistent the raters are? I'm just trying to make sure I can
>> replicate the problem, because when I tried to fit one model I got one
>> answer and another blew up.
>>
>> I ask because I'm writing a paper on ICC estimation and have been
>> considering problems very similar to the one you have here.
>>
>>
>> On Tue, Feb 26, 2013 at 2:35 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>>> A scatter plot matrix is instructive.
>>>
>>> Warning: destroys your data.
>>>
>>> reshape wide Score rank , i(Application) j(Rator)
>>> graph matrix Score?
>>>
>>> #4 really is oddball.
>>>
>>> Another interesting plot is
>>>
>>> parplot Score?, tr(raw)
>>>
>>> where -parplot- must be installed from SSC first.
>>>
>>> Nick
>>>
>>> On Tue, Feb 26, 2013 at 7:24 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>>> Hi Rebecca,
>>>> Thank you for your help.  As a clarification:
>>>> We used scores originally, but Rater 4's scores were all very low.
>>>> Thus, when we ranked them, there were a lot of ties.  As seen below, 8
>>>> of the 11 apps got a rank of "2" by rater.
>>>> Lenny
>>>>
>>>> Application Rator Score rank
>>>> 5 1 2 1
>>>> 7 1 5 2
>>>> 2 1 6 3
>>>> 9 1 6 3
>>>> 11 1 7 4
>>>> 6 1 7 4
>>>> 8 1 11 5
>>>> 3 1 13 6
>>>> 4 1 16 7
>>>> 10 1 17 8
>>>> 1 1 18 9
>>>> 6 2 1 1
>>>> 5 2 2 2
>>>> 11 2 3 3
>>>> 7 2 3 3
>>>> 4 2 5 4
>>>> 1 2 7 5
>>>> 8 2 8 6
>>>> 2 2 9 7
>>>> 3 2 10 8
>>>> 10 2 12 9
>>>> 9 2 12 9
>>>> 5 3 2 1
>>>> 2 3 5 2
>>>> 7 3 6 3
>>>> 6 3 6 3
>>>> 9 3 6 3
>>>> 11 3 7 4
>>>> 8 3 11 5
>>>> 3 3 13 6
>>>> 4 3 15 7
>>>> 10 3 16 8
>>>> 1 3 17 9
>>>> 7 4 0 1
>>>> 1 4 1 2
>>>> 9 4 1 2
>>>> 6 4 1 2
>>>> 8 4 1 2
>>>> 4 4 1 2
>>>> 5 4 1 2
>>>> 3 4 1 2
>>>> 11 4 1 2
>>>> 2 4 2 3
>>>> 10 4 3 4
>>>>
>>>> On Tue, Feb 26, 2013 at 9:54 AM, Rebecca Pope <rebecca.a.pope@gmail.com> wrote:
>>>>> Lenny,
>>>>> I was just addressing your syntax error, not your underlying data
>>>>> issues. Why would you expect a ratio to increase when you've made the
>>>>> numerator 0? If you are getting an ICC close to 0, you should think
>>>>> about what that is telling you about your data.
>>>>>
>>>>> If you look at e.g. judges.dta (example for -icc-), you'll see that
>>>>> the results for the ICC is the same regardless of the method that you
>>>>> use.
>>>>>
>>>>> webuse judges
>>>>> icc rating target judge, mixed
>>>>> xtmixed rating i.judge || _all: R.target, reml var
>>>>> nlcom exp(_b[lns1_1_1:_cons])^2/(exp(_b[lnsig_e:_cons])^2+exp(_b[lns1_1_1:_cons])^2)
>>>>>
>>>>> The two ICCs are nearly equal (to 6 decimal places). Using -xtmixed-
>>>>> will never give you a negative value though.
>>>>>
>>>>> An aside: "I'm using the ranks (within an individual) instead of the
>>>>> actual scores."
>>>>>
>>>>> If you are using rankings (1-11 presumably) within individual rather
>>>>> than actual scores it isn't clear to me how rater 4 could be "off the
>>>>> charts" regardless of actual scores assigned. By converting scores to
>>>>> rankings, you've wiped out the correlation of scores within rater. You
>>>>> seem to be interested instead in how e.g. app 1 is rated by all 4
>>>>> raters (correlation within app). If raters 1, 2, 3 all give it a score
>>>>> of 1 (their preferred app) & rater 4 gives it a 6, you don't want to
>>>>> drop that info. That is what you are analyzing.
>>>>>
>>>>> Regards,
>>>>> Rebecca
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 26, 2013 at 10:16 AM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>>>>> Thanks Rebecca,
>>>>>> With that code, I get the same problem when I eliminate one rater.
>>>>>>
>>>>>> the var(rater) goes to zero, which makes my ICC 0, rather go up to a
>>>>>> higher number as I expected.
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Rebecca Pope <rebecca.a.pope@gmail.com>
>>>>>> Date: Tue, Feb 26, 2013 at 7:08 AM
>>>>>> Subject: Re: st: reliability with -icc- and -estat icc-
>>>>>> To: statalist@hsphsun2.harvard.edu
>>>>>>
>>>>>>
>>>>>> Lenny,
>>>>>> I don't think you've got the correct syntax for -xtmixed- if you are
>>>>>> trying to duplicate ANOVA results, which is the type of analysis that
>>>>>> -icc- appears to conduct (documentation is still limited, so I won't
>>>>>> swear to anything).
>>>>>>
>>>>>> Use this syntax for -xtmixed-:
>>>>>> xtmixed rank i.Application || _all: R.Rater, reml var
>>>>>>
>>>>>> -estat icc- is not a valid post-estimation command after this
>>>>>> specification. However, you can just use the definition that ICC =
>>>>>> Var(Rater)/(Var(Rater)+Var(Residual)).
>>>>>>
>>>>>> You might also want to take a look at
>>>>>> http://www.ats.ucla.edu/stat/stata/faq/xtmixed.htm which will give you
>>>>>> instructions for using -xtmixed- to conduct ANOVA-type analyses (using
>>>>>> Stata 10, so you'll need to modify somewhat).
>>>>>>
>>>>>> Regards,
>>>>>> Rebecca
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 25, 2013 at 10:56 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
>>>>>>> I have 4 raters that gave a score of 0-100 on 11 smartphone applications.
>>>>>>> The data is skewed right, as they all got low scores.  I'm using the
>>>>>>> ranks (within an individual) instead of the actual scores.  I want to
>>>>>>> know the correlation in ranking between the different raters.
>>>>>>>
>>>>>>> I've tried the two commands:
>>>>>>>
>>>>>>> -xtmixed rank Application || Rater: , reml
>>>>>>> -estat icc
>>>>>>>
>>>>>>> (icc=0.19)
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>> -icc rank Rater Application, mixed consistency
>>>>>>>
>>>>>>> (icc=0.34)
>>>>>>>
>>>>>>> They give me two different answers. Which one is correct?
>>>>>>>
>>>>>>>
>>>>>>> Next, we found out that rater 4 was off the charts, and we want to
>>>>>>> eliminate her and rerun the analysis. When we do this we get wacky
>>>>>>> ICCs.  In the first method we get an ICC of 2e-26.  In the 2nd method
>>>>>>> (-icc), we get -.06.  Eliminating any of the other raters gives us
>>>>>>> ICCs close to the original ICC.  Why are we getting such a crazy
>>>>>>> number when we eliminate this 4th rater?
>>>>>>>
>>>>>>>
>>>>>>> I'm guessing this might be instability in the model, but I'm not sure
>>>>>>> how to get around it.
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/