Re: st: composite kappa & intercenter variability in kappas

[Constantine Daskalakis <C_Daskalakis@mail.jci.tju.edu>]

At 01:50 PM 8/17/2005, Richard_Lenhardt@rush.edu wrote:
> Hello
>
> We submitted a manuscript comparing two observers measuring 10 dicotomous
> variables.  We measured kappa, to assess concordance between observers.  A
> reviewer asked that we calculate an overall value (composite) of the
> kappas, for all 10 variables.   (Fleiss, Stat Methods for Rates and
> Proportions, 2nd or 3rd edition).
>
> I couldn't find such a function in stata (overall kappa).  Would there be
> another way to calculate this other than by hand?
>
> Also, the study was completed at 5 different hospitals.  The reviewer
> asked that we take into account hospital to hospital variation when
> calculating kappas.  I wasn't aware that kappas could be adjusted for the
> 5 hospitals, other than generating a large matrix of kappas corresponding
> to the different hospitals.  Would there be a way to take into account the
> variation in agreement among the hospitals when estimating the kappas?
>
> Thank you!
>
> Richard Lenhardt, MD, MPH
> Assistant Professor of Medicine
> Division of Pulmonary and Critical Care Medicine
> Rush University Medical Center
> Chicago, IL

Hospital can be used as a stratification factor.
You say you have 10 dichotomous variables that are being rated. This is 
what you can do.

Take variable Y1 first. Compute stratum-specific kappas (for each hospital 
separately). Then, compute a weighted-average combined kappa. Whether 
that's reasonable depends on whether you are estimating the same kappa 
parameter in all the strata -- eyeballing and a homogeneity test might help 
here. In flavor, all this is like stratification and computation of an 
adjusted MH odds ratio. I don't have Fleiss in front of me but the 
specifics are in there.

Then do the same thing for the rest of your variables, Y2, ..., Y10.

You now have 10 "adjusted" kappas (controlling for hospital).

Should you compute an overall adjusted kappa across all 10 ratings? Maybe 
yes, maybe no.

Could you do it? In principle, you could use the same procedure to get a 
weighted kappa from all those 10 kappas. The trouble is that the individual 
kappas are correlated beyond the fact that you are using the same raters -- 
you are also probably using the same patients. I think the only feasible 
way to get a handle on such a beast is through the bootstrap.

Hope this helps.

CD




The documents accompanying this transmission may contain confidential 
health or business information. This information is intended for the use of 
the individual or entity named above. If you have received this information 
in error, please notify the sender immediately and arrange for the return 
or destruction of these documents.

________________________________________________________________
Constantine Daskalakis, ScD
Assistant Professor,
Thomas Jefferson University, Division of Biostatistics,
211 S. 9th St., Suite 602, Philadelphia, PA 19107
   Tel: 215-955-5695
   Fax: 215-503-3804
   Email: c_daskalakis@mail.jci.tju.edu
   Webpage: http://www.jefferson.edu/clinpharm/bio/  
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/