RE: st: Interrater Agreement and Kap Command

["Visintainer, Paul" <Paul.Visintainer@baystatehealth.org>]

You can try -icc23-, which is downloadable in Stata (type: findit icc23).  It will compute type 2 and type 3 intra-class correlations, using error components derived from ANOVA.

Paul

________________________________________________
Paul F. Visintainer, PhD
Baystate Medical Center
Springfield, MA 01199


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Frank Gallo
Sent: Tuesday, July 13, 2010 8:28 PM
To: statalist
Subject: Re: st: Interrater Agreement and Kap Command

Hi Morten,

Thank you for your response. Let me clarify my situation. The ratings
are interval-like. The data file is in the following format: columns
(raters, n = 94) and rows (variables, n = 49). The raters gave
responses that ranged from 1 to 9 on the variables. I am looking for
something similar to the ICC (model: two-way mixed, alpha; type:
consistency) available in SPSS. I hope this helps.

Best,
Frank


On Tue, Jul 13, 2010 at 5:53 PM, Morten Hesse <mh@crf.au.dk> wrote:
> Dear Frank
> You wouldn't want to use the ICC if your values are not interval scale. What
> I know of SPSS that would be the ICC for an interval scale. There is a
> similar statistic for logistic regression (the rho statistic), but in my
> experience, in cases where kappa values suggest moderate agreement between
> any two observations, the rho statistic derived from xtlogit or MLE logistic
> regression in other software will show very high values of agreement, and I
> do not believe that they represent reliability estimates that are realistic,
> although I do not have the expertise to say so (any thoughts, anyone else).
> Therefore, you would want to use Fleiss's kappa. I am not sure if that or a
> similar statistic is available for Stata.
> Percent agreement should be fairly easy to estimate using an excell
> spreadsheet.
> One question: If you say 49 variables, would that refer to different
> outcomes pertaining to the same units of observation?
> Greetings
> Morten
>
> Frank Gallo skrev:
>>
>> Dear All:
>>
>> I am a beginner with Stata, and I prefer to use Stata instead of SPSS
>> to address the following situation. I want to evaluate interrater
>> agreement. I have more than two raters (94), more than two ratings (49
>> variables), and varying number of raters (some missing values on the
>> variables). The data are in the form to use the kap command, however,
>> there is no statistic for testing kappa > 0 and no results reported
>> regarding the percentage of agreement. Is there some alternative that
>> I can use in Stata to obtain these results? I looked through the
>> archives and could not find one. SPSS offers the ICC option. I
>> appreciate any thoughts.
>>
>> Best,
>> Frank
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

----------------------------------------------------------------------
Please view our annual report at http://baystatehealth.org/annualreport
 

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain confidential and privileged information for the use of the designated recipients named above. If you are not the intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please reply to the sender immediately or by telephone at 413-794-0000 and destroy all copies of this communication and any attachments. For further information regarding Baystate Health's privacy policy, please visit our Internet site at http://baystatehealth.org.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/