No long ago I posted a question regarding how to pool individual kappa
statistics (each from the comparison between one rater and the answer
key scorer); I have 7 raters, then I need to pool these 7 kappas.
I am thinking to pool them using meta-analytical techniques, like
inverse weighting on the SD. Does anyone knows if such approach is
statistically valid?
Any advice would be very much appreciated.
Eduardo Nunez, MD, MPH
Valencia, Spain
---------- Forwarded message ----------
From: Eduardo Nunez <[email protected]>
Date: Sat, Feb 6, 2010 at 9:47 PM
Subject: st: Random effect Kappa?
To: [email protected]
Dear Statalisters:
I am seeking your guidance on how to perform either random effect or
accounting for autocorrelation when estimating kappa statistics. I
need to figure out a way to perform this statistical analysis in
Stata. If not possible, I would appreciate any other software capable
of it.
These are the details and challenge I have before me:
1. 7 blinded scorers (dermatologists, plastic surgeon)
2. 188 photographs representing four facial compartments
3. 10 possible scores (0-9, none to severe laxity) for each photo by
each blinded scorer
The data for the first scorer looks like:
Scorer1
pte_id photo_id laxity_score(0-9) facial_region(1-4)
lead_author_score
1 1 8 3 7
1 25 3 1 3
2 4 0 4 0
and so on....
For the rest of the scorer (2 to 7), I have similar data structure.
Outcomes:
1) Agreement (kappa) of each scorer with the lead author score (to
compare each of the blinded scorer's results against an "answer key"
provided by the lead author).
2) Agreement (kappa) for the comparison between each scorer (1 vs 2, 1
vs 3, 2 vs 1, etc).
2) A summary kappa integrating each kappa from 1),
3) A summary kappa integrating each kappa from 2),
Questions:
1) Because one patient can contributes to several photos and to
different facial regions, I assume measures from the same pte are
likely to be correlated. How should I adjust for that? random effect?
2) Which is the best way to pool individual kappas into a grand summary kappa?
Any advice would be very much appreciated.
Eduardo Nunez, MD, MPH
Valencia, Spain
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
