RE: st: RE: Kappa query - unequal distribution of scores

["Downs, Sara" <Sara.Downs@ahvla.gsi.gov.uk>]

Hi Nick,

Thanks for the advice which was helpful.

I also found these references that suggest that if the marginal totals are unbalanced the kappa statistic may provide an under or over estimate of association:
Feinstein & Cicchetti (1990) J Clin Epidemiol, vol 43, no 6, 543-549  
Cicchetti & Feinstein (1990) J Clin Epidemiol, vol 43, no 6, 551-558
Sara


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: 22 September 2011 11:44
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: RE: Kappa query - unequal distribution of scores

. bootstrap r(kappa), nodots reps(10000): kap rev1 rev2

[...]

Bootstrap results                               Number of obs      =        83
                                                Replications       =     10000

      command:  kap rev1 rev2
        _bs_1:  r(kappa)

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------
-------------+------
       _bs_1 |    .020649   .0983171     0.21   0.834     -.172049     .213347
------------------------------------------------------------------------------

The s.e. is a bit higher. In this case, as kappa is almost 0, it is pretty much immaterial what s.e. you get.

Nick

On Thu, Sep 22, 2011 at 10:36 AM, Nick Cox <njcoxstata@gmail.com> wrote:
> I'd think about bootstrapping here.

> On Thu, Sep 22, 2011 at 10:06 AM, Downs, Sara 
> <Sara.Downs@ahvla.gsi.gov.uk> wrote:
>> Hi,
>>
>> I am trying to calculate agreement between 2 distributions of scores.
>> There are 3 possible scores: One score predominates and the other 2 
>> are extremely rare :
>>
>>  tab rev1 rev2
>>
>>           |               rev2
>>      rev1 |         0          1          2 |     Total
>> -----------+---------------------------------+----------
>>         0 |         1          6          0 |         7
>>         1 |        13         57          3 |        73
>>         2 |         0          2          1 |         3
>> -----------+---------------------------------+----------
>>     Total |        14         65          4 |        83
>>
>>
>> .         kap rev1 rev2
>>
>>             Expected
>> Agreement   Agreement     Kappa   Std. Err.         Z      Prob>Z
>> -----------------------------------------------------------------
>>  71.08%      70.47%     0.0206     0.0852       0.24      0.4042
>>
>>
>> I understand that the usual kappa statistic is not reliable in this 
>> situation .   Is there any way in STATA that I can adjust for the 
>> high expected frequency. I am using STATA 10.1.
>>
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
Animal Health and Veterinary Laboratories Agency (AHVLA)
 
This email and any attachments is intended for the named recipient only.
If you have received it in error you have no authority to use, disclose, 
store or copy any of its contents and you should destroy it and inform the sender.
Whilst this email and associated attachments will have been checked for known viruses whilst within AHVLA systems we can accept no responsibility once it has left our systems.
Communications on AHVLA computer systems may be monitored and/or recorded to secure the effective operation of the system and for other lawful purposes.


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/