Bernadette
You can calculate an analytical confidence interval for kappa for
multiple raters and a binary rating using the method of Zou & Donner
(Biometrics 2004) - the commands -kappci- and -kappaci- to do this
calculation (for -kap- and -kappa- formatted data, respectively) are
included in my package -kaputil- available from SSC (-ssc describe
kaputil-).
Alternatively, you may want to take a look at the command -kapci-, from
Stata J 4-4 (-findit kapci-), which does bootstrap cis for kappa,
although my recollection is that the data need to be in the format for
-kap- for this one.
You should note, though, that both -kap- and -kappa- require one row per
subject rather than one row per rater. You can get from one form to the
other with two -reshape-s, e.g. suppose you have a variable rater taking
values 1-149 and 14 variables rating1-rating14 for the ratings of the 14
videos...
reshape long rating, i(rater) j(video)
reshape wide rating, i(video) j(rater)
Now you have a variable video taking values 1-14 and 149 variables
rating1-rating149 for the ratings of the 149 raters, and you can use
-kap- (or -kappci- or -kapci- or do your own bootstrapping).
Hope this helps
David
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of
[email protected]
Sent: 20 June 2006 05:18
To: [email protected]
Subject: st: bootstrap confidence intervals for multiple rater kappa
I have a dataset where 149 raters have classified 14 video scenarios
into a binary (1,0) classification system on two occasions, once before
and once after an education intervention. I'm interested in seeing
whether the education intervention improved the agreement of the raters
in classifying the video scenarios. I can use the "kappa" command on a
table summarising this dataset to calculate the kappa statistic across
the 14 scenarios on each occasion. I would now like to calculate
bootstrapped confidence intervals for these and also compare kappa
statistics between the two assessment occasions.
However, I would like the bootstrap procedure to resample raters rather
than the 14 video scenarios. Can anyone tell me how I can calculate
kappa in the case of more than two raters using a dataset where each
rater and their classifications of the video scenarios are presented as
an individual row in the dataset so that these can be resampled when I
perform the bootstrap procedure?
Bernadette Massey
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
