Re: st: Interrater Agreement and Kap Command

[Morten Hesse <mh@crf.au.dk>]

Dear Frank
You wouldn't want to use the ICC if your values are not interval scale. 
What I know of SPSS that would be the ICC for an interval scale. There 
is a similar statistic for logistic regression (the rho statistic), but 
in my experience, in cases where kappa values suggest moderate agreement 
between any two observations, the rho statistic derived from xtlogit or 
MLE logistic regression in other software will show very high values of 
agreement, and I do not believe that they represent reliability 
estimates that are realistic, although I do not have the expertise to 
say so (any thoughts, anyone else). Therefore, you would want to use 
Fleiss's kappa. I am not sure if that or a similar statistic is 
available for Stata.
Percent agreement should be fairly easy to estimate using an excell 
spreadsheet.
One question: If you say 49 variables, would that refer to different 
outcomes pertaining to the same units of observation?
Greetings
Morten

Frank Gallo skrev:
> Dear All:
>
> I am a beginner with Stata, and I prefer to use Stata instead of SPSS
> to address the following situation. I want to evaluate interrater
> agreement. I have more than two raters (94), more than two ratings (49
> variables), and varying number of raters (some missing values on the
> variables). The data are in the form to use the kap command, however,
> there is no statistic for testing kappa > 0 and no results reported
> regarding the percentage of agreement. Is there some alternative that
> I can use in Stata to obtain these results? I looked through the
> archives and could not find one. SPSS offers the ICC option. I
> appreciate any thoughts.
>
> Best,
> Frank
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>   
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/