RE: st: AW: Non-reported s.e. for kappa

["Nick Cox" <n.j.cox@durham.ac.uk>]

On a quite different note, I am curious. How much practical use is a single kappa measure with multiple raters? If this were my problem I would be much more interested in looking at the fine structure of disagreement, e.g. whether raters fall into distinct groups, using some sort of multivariate method. 

Imagine instead of a correlation matrix a single super-correlation.... It wouldn't be much use, so far as I can see. 

Nick 
n.j.cox@durham.ac.uk 

Steve Samuels

You are welcome, Karen.  I'm flattered to be mistaken for Martin.
With n = 26, you might also try -bootstrap-.


On Wed, Apr 14, 2010 at 12:20 PM, Karen Wright <Karen.Wright@icr.ac.uk> wrote:

> Deepest apologies, thanks Steve for the jackknife suggestion.
> Although thanks also to Martin for your input!

>>>> Steve Samuels <sjsamuels@gmail.com> 14/04/2010 17:18 >>>
> Correction, to get the pseudo-values _jk_1:
>
> jackknife r(kappa), mse keep: kap id*
> ci _jk_1      // First order unbiased estimates
>
>
> Steve
> On Wed, Apr 14, 2010 at 12:03 PM, Steve Samuels <sjsamuels@gmail.com>
> wrote:
>> Martin-
>> For Karen's set up (idj contains the subject rating for  rater j),
>> there is no official example that I can see. p615 shows counts of
>> ratings, with each variable being a rating category.  Also, for
> p615,,
>>  the command should be "kappa",  not "kap", because the data are
>> counts of assignments to three categories by five raters. This was
>> terrible  syntax decision: depending on the use of "kap" or "kappa"
> to
>> distinguish between two data setups.
>>
>> One suggestion for a standard error:
>> *************************************************
>> version 10.0
>> clear
>>  input case id2 id3 id6 id8 id9 id10
>>  1 3 3 3 3 3 3
>>  2 1 1 1 1 1 1
>>  3 2 2 1 3 1 1
>>  4 3 3 3 3 3 3
>>  5 1 1 1 1 1 1
>>  end
>> kap id*
>> jackknife r(kappa), mse: kap id*
>> ci _jk_1      // First order unbiased estimates
>> *************************************
>>
>>  n = 5 subjects doesn't seem like enough to draw many conclusions.
>> It's interesting that the raters disagreed for only one subject.
>>
>> Steve
>>
>> Steve
>>
>> On Wed, Apr 14, 2010 at 10:52 AM, Martin Weiss <martin.weiss1@gmx.de>
> wrote:
>>>
>>> <>
>>>
>>> You do not need an imaginary dataset, as the official example does
> not
>>> return the se, either:
>>>
>>>
>>> *************
>>> webuse p615, clear
>>> kap cat*
>>> ret li
>>> *************
>>>
>>> The help file could make this fact clear, I think. BTW, a command
> reporting
>>> a whole table inevitably runs into the problem which statistics it
> should
>>> return. Just see what -summ- with a -varlist- hands back: The
> statistics for
>>> the last variable in that -varlist-. The alternative would be to
> return a
>>> -matrix-, of course...
>>>
>>>
>>> HTH
>>> Martin
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: owner-statalist@hsphsun2.harvard.edu
>>> [mailto:owner-statalist@hsphsun2.harvard.edu] Im Auftrag von Karen
> Wright
>>> Gesendet: Mittwoch, 14. April 2010 16:38
>>> An: statalist@hsphsun2.harvard.edu
>>> Betreff: st: Non-reported s.e. for kappa
>>>
>>> Dear All,
>>> I have been working with kappa statistics and I have come across a
>>> peculiarity that I don't fully understand.  Searching the help files
> and the
>>> statalist archive did not shed any light on the matter
> unfortunately.
>>> After running a simple composite kappa on multiple readers (3
> categories)  I
>>> then type -di r(se)- and obtain nothing!  The results have not been
> cleared
>>> from memory as typing -di r(kappa)- displays the kappa value.  Does
> anyone
>>> know why this is?
>>> I am aware that test statistics cannot be calculated when the number
> of
>>> ratings per subject vary.  Is there something similar happening
> here?!
>>>
>>> I have shown below a snippet of the data and the commands used.
> Snippet of
>>> data contains the diagnosis (=1,2, or 3) for 6 readers (id2-id10) on
> 5
>>> cases.  [Done using Stata10 on Windows.]
>>>
>>> Thank you for taking the time to read this,
>>> Karen
>>>
>>> *************************************************
>>> version 10.0
>>>
>>> input case id2 id3 id6 id8 id9 id10
>>> 1 3 3 3 3 3 3
>>> 2 1 1 1 1 1 1
>>> 3 2 2 1 3 1 1
>>> 4 3 3 3 3 3 3
>>> 5 1 1 1 1 1 1
>>> end
>>>
>>> kap id*
>>> di r(se)
>>> di r(kappa)
>>>
>>> *************************************************
>>>
>>>
>>>
>>> The Institute of Cancer Research: Royal Cancer Hospital, a
> charitable
>>> Company Limited by Guarantee, Registered in England under Company
> No. 534147
>>> with its Registered Office at 123 Old Brompton Road, London SW7
> 3RP.
>>>
>>> This e-mail message is confidential and for use by the addressee
> only.  If
>>> the message is received by anyone other than the addressee, please
> return
>>> the message to the sender by replying to it and then delete the
> message from
>>> your computer and network.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/