t 01:50 PM 8/17/2005, [email protected] wrote:
Hello
We submitted a manuscript comparing two observers measuring 10 dicotomous
variables. We measured kappa, to assess concordance between observers. A
reviewer asked that we calculate an overall value (composite) of the
kappas, for all 10 variables. (Fleiss, Stat Methods for Rates and
Proportions, 2nd or 3rd edition).
I couldn't find such a function in stata (overall kappa). Would there be
another way to calculate this other than by hand?
Also, the study was completed at 5 different hospitals. The reviewer
asked that we take into account hospital to hospital variation when
calculating kappas. I wasn't aware that kappas could be adjusted for the
5 hospitals, other than generating a large matrix of kappas corresponding
to the different hospitals. Would there be a way to take into account the
variation in agreement among the hospitals when estimating the kappas?
Thank you!
Richard Lenhardt, MD, MPH
Assistant Professor of Medicine
Division of Pulmonary and Critical Care Medicine
Rush University Medical Center
Chicago, IL
Hospital can be used as a stratification factor.
You say you have 10 dichotomous variables that are being rated. This is
what you can do.
Take variable Y1 first. Compute stratum-specific kappas (for each hospital
separately). Then, compute a weighted-average combined kappa. Whether
that's reasonable depends on whether you are estimating the same kappa
parameter in all the strata -- eyeballing and a homogeneity test might
help here. In flavor, all this is like stratification and computation of
an adjusted MH odds ratio. I don't have Fleiss in front of me but the
specifics are in there.
Then do the same thing for the rest of your variables, Y2, ..., Y10.
You now have 10 "adjusted" kappas (controlling for hospital).
Should you compute an overall adjusted kappa across all 10 ratings? Maybe
yes, maybe no.
Could you do it? In principle, you could use the same procedure to get a
weighted kappa from all those 10 kappas. The trouble is that the
individual kappas are correlated beyond the fact that you are using the
same raters -- you are also probably using the same patients. I think the
only feasible way to get a handle on such a beast is through the
bootstrap.
Hope this helps.
CD
The documents accompanying this transmission may contain confidential
health or business information. This information is intended for the use
of the individual or entity named above. If you have received this
information in error, please notify the sender immediately and arrange for
the return or destruction of these documents.
________________________________________________________________
Constantine Daskalakis, ScD
Assistant Professor,
Thomas Jefferson University, Division of Biostatistics,
211 S. 9th St., Suite 602, Philadelphia, PA 19107
Tel: 215-955-5695
Fax: 215-503-3804
Email: [email protected]
Webpage: http://www.jefferson.edu/clinpharm/bio/
CD,
Thank you for your help, now I understand what to do with stata. This
brings me to a follow up question (for anyone):
To calculate an "adjusted" kappa, based on 5 hospitals, according to
Fleiss, one would sum the hospital kappas, weighted by the inverse of the
square of the hospital kappa standard errors.
adjusted kappa for given item = SUM kappahosp(i) /
SUM 1/se(kappahosp(i))^2
se(kappahosp(i))^2 /
I'm trying to write a do program that enables this with the following
variables:
Study ID HospID A1 B1 A2 B2 ....
1 1 Y Y Y N
2 1 N N Y Y
and so on
statsby "kap A1 B1" k1=r(kappa) se1=R(se), by(hospid)saving(temp1) replace
use temp1, clear
gen adjustedkappa=
I can't seem to figure out how to generate a variable "adjustedkappa"
whereby the variable is the sum of the 5 kappas calculated earlier for
each individual hospital.
I'd like to do the same for the the std error term and the 95% confidence
interval. The std error = square root(SUM 1/se(kappahosp(i))^2)
Finally, is there a way to program this, so that I could simply write:
statsby "kap X Y" k=r(kappa) se=R(se), by(hospid)saving(temp1) replace
so I don't have to re-write a statsby line for each item that was observed
(A1/B2, A2/B2, A3/B3, ...)
Thanks so much!
Richard Lenhardt, MD, MPH
Assistant Professor of Medicine
Division of Pulmonary and Critical Care Medicine
Rush University Medical Center
Chicago, IL
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
