Re: st: RE: Interobserver agreement for continuous variables

[Roger Newson <roger.newson@kcl.ac.uk>]

At 21:43 17/10/02 -0700, Nitin wrote:
> Hi Nick and Scott,
> Thank you for the response. But the concord and other
> tests contained in it are only for 2 observers/raters.
> I am looking for something that calculates
> interobserver agreement for a continuous variable WHEN
> THERE ARE MORE THAN 3 RATERS AND THE number of raters
> vary for each subject ?
> I look forward to hearing from you.
> Nitin
I personally would use a variation on the intra-class Kendall's tau-a, 
which can be calculated (with confidence limits) using my -somersd- 
package, downloadable from SSC. (Type -ssc describe somersd- or -net search 
somersd- within Stata to find how to download.) First, create a data set 
with 1 obs for every ordered pair of measurements on the same subject by 
different observers. Such a data set might have variables as follows:

subjid       Subject ID
obsid1      First observer ID
obsid2      Second observer ID
y1            First observer rating
y2            Second observer rating

Such a data set might be created using the -joinby- package. Note that each 
combination of Observer A, Observer B and Patient X should appear twice in 
this data set, once with Observer A as first observer and Observer B as 
second observer, and once with Observer B as first observer and Observer A 
as second observer. Once you have created this data set, type

somersd y1 y2,taua tdist cluster(subjid)

and Stata will calculate confidence limits for a Kendall tau-a between -y1- 
and -y2-, clustered by -subjid-. This tau-a is a difference between 2 
probabilities, namely the probability of concordance (two raters order two 
subjects in the same way) and the probability of discordance (two raters 
order two subjects in the opposite way). For instance, if Kendall's tau-a 
is 0.70, then this means that, if 2 observers are given 2 randomly chosen 
subjects and asked which one to rate more highly, then the observers are 
70% more likely to agree than to disagree.

For more information on Kendall's tau-a, a possible reference is Newson 
(2002), which in turn has many more references.

I hope this helps.

Roger

References

Newson R. Parameters behind "non-parametric" statistics: Kendall's tau, 
Somers' D and median differences. The Stata Journal 2002; 2(1): 45-64.


--
Roger Newson
Lecturer in Medical Statistics
Department of Public Health Sciences
King's College London
5th Floor, Capital House
42 Weston Street
London SE1 3QD
United Kingdom

Tel: 020 7848 6648 International +44 20 7848 6648
Fax: 020 7848 6620 International +44 20 7848 6620
  or 020 7848 6605 International +44 20 7848 6605
Email: roger.newson@kcl.ac.uk

Opinions expressed are those of the author, not the institution.

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/