Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: st: AW: Non-reported s.e. for kappa
From
"Karen Wright" <[email protected]>
To
<[email protected]>
Subject
RE: st: AW: Non-reported s.e. for kappa
Date
Thu, 15 Apr 2010 09:22:47 +0100
I am aware of this fact and the readers are spit into different groups
based on experience, how many cases rated, expert, etc, and to have a
weighted kappa for each group to compare the level of agreement over
different years (circulations of an EQA scheme). I merely simplified
the example for the purpose of asking about the missing se from the
kappa command. I am doing analysis under instruction and have been told
to use kappa and develop a method of applying weighted kappa to multiple
readers, despite said disadvantages fo the kappa statistic itself.
Thank you all for your advice.
>>> "Nick Cox" <[email protected]> 14/04/2010 18:39 >>>
On a quite different note, I am curious. How much practical use is a
single kappa measure with multiple raters? If this were my problem I
would be much more interested in looking at the fine structure of
disagreement, e.g. whether raters fall into distinct groups, using some
sort of multivariate method.
Imagine instead of a correlation matrix a single super-correlation....
It wouldn't be much use, so far as I can see.
Nick
[email protected]
Steve Samuels
You are welcome, Karen. I'm flattered to be mistaken for Martin.
With n = 26, you might also try -bootstrap-.
On Wed, Apr 14, 2010 at 12:20 PM, Karen Wright <[email protected]>
wrote:
> Deepest apologies, thanks Steve for the jackknife suggestion.
> Although thanks also to Martin for your input!
>>>> Steve Samuels <[email protected]> 14/04/2010 17:18 >>>
> Correction, to get the pseudo-values _jk_1:
>
> jackknife r(kappa), mse keep: kap id*
> ci _jk_1 // First order unbiased estimates
>
>
> Steve
> On Wed, Apr 14, 2010 at 12:03 PM, Steve Samuels
<[email protected]>
> wrote:
>> Martin-
>> For Karen's set up (idj contains the subject rating for rater j),
>> there is no official example that I can see. p615 shows counts of
>> ratings, with each variable being a rating category. Also, for
> p615,,
>> the command should be "kappa", not "kap", because the data are
>> counts of assignments to three categories by five raters. This was
>> terrible syntax decision: depending on the use of "kap" or "kappa"
> to
>> distinguish between two data setups.
>>
>> One suggestion for a standard error:
>> *************************************************
>> version 10.0
>> clear
>> input case id2 id3 id6 id8 id9 id10
>> 1 3 3 3 3 3 3
>> 2 1 1 1 1 1 1
>> 3 2 2 1 3 1 1
>> 4 3 3 3 3 3 3
>> 5 1 1 1 1 1 1
>> end
>> kap id*
>> jackknife r(kappa), mse: kap id*
>> ci _jk_1 // First order unbiased estimates
>> *************************************
>>
>> n = 5 subjects doesn't seem like enough to draw many conclusions.
>> It's interesting that the raters disagreed for only one subject.
>>
>> Steve
>>
>> Steve
>>
>> On Wed, Apr 14, 2010 at 10:52 AM, Martin Weiss
<[email protected]>
> wrote:
>>>
>>> <>
>>>
>>> You do not need an imaginary dataset, as the official example does
> not
>>> return the se, either:
>>>
>>>
>>> *************
>>> webuse p615, clear
>>> kap cat*
>>> ret li
>>> *************
>>>
>>> The help file could make this fact clear, I think. BTW, a command
> reporting
>>> a whole table inevitably runs into the problem which statistics it
> should
>>> return. Just see what -summ- with a -varlist- hands back: The
> statistics for
>>> the last variable in that -varlist-. The alternative would be to
> return a
>>> -matrix-, of course...
>>>
>>>
>>> HTH
>>> Martin
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: [email protected]
>>> [mailto:[email protected]] Im Auftrag von Karen
> Wright
>>> Gesendet: Mittwoch, 14. April 2010 16:38
>>> An: [email protected]
>>> Betreff: st: Non-reported s.e. for kappa
>>>
>>> Dear All,
>>> I have been working with kappa statistics and I have come across a
>>> peculiarity that I don't fully understand. Searching the help
files
> and the
>>> statalist archive did not shed any light on the matter
> unfortunately.
>>> After running a simple composite kappa on multiple readers (3
> categories) I
>>> then type -di r(se)- and obtain nothing! The results have not
been
> cleared
>>> from memory as typing -di r(kappa)- displays the kappa value.
Does
> anyone
>>> know why this is?
>>> I am aware that test statistics cannot be calculated when the
number
> of
>>> ratings per subject vary. Is there something similar happening
> here?!
>>>
>>> I have shown below a snippet of the data and the commands used.
> Snippet of
>>> data contains the diagnosis (=1,2, or 3) for 6 readers (id2-id10)
on
> 5
>>> cases. [Done using Stata10 on Windows.]
>>>
>>> Thank you for taking the time to read this,
>>> Karen
>>>
>>> *************************************************
>>> version 10.0
>>>
>>> input case id2 id3 id6 id8 id9 id10
>>> 1 3 3 3 3 3 3
>>> 2 1 1 1 1 1 1
>>> 3 2 2 1 3 1 1
>>> 4 3 3 3 3 3 3
>>> 5 1 1 1 1 1 1
>>> end
>>>
>>> kap id*
>>> di r(se)
>>> di r(kappa)
>>>
>>> *************************************************
>>>
>>>
>>>
>>> The Institute of Cancer Research: Royal Cancer Hospital, a
> charitable
>>> Company Limited by Guarantee, Registered in England under Company
> No. 534147
>>> with its Registered Office at 123 Old Brompton Road, London SW7
> 3RP.
>>>
>>> This e-mail message is confidential and for use by the addressee
> only. If
>>> the message is received by anyone other than the addressee, please
> return
>>> the message to the sender by replying to it and then delete the
> message from
>>> your computer and network.
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.
This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/