st: RE: RE: reliability with ordinal data-Kendall's w?

["Ploutz-Snyder, Robert (JSC-SK)[USRA]" <robert.ploutz-snyder-1@nasa.gov>]

Thank you Nick,
I use Somers' d in other instances, but thought it not clear that it is appropriate for reliability analysis?  

I arrive at this as:  In the case with two continuously scaled variables (A,B) we use Pearson's r to describe the association (correlation), however if we are really talking about the SAME thing measured with two devices or by two raters (Y1,Y2) we would rely on ICC or Kappa statistics to understand reliability.  

Perhaps I am being over simplistic, but in my view, Somers' d would be an appropriate tool for the first situation above (A,B), but perhaps not the second (Y1,Y2).

My second reason is that Somers' d is closely related to Kendall's tau a, except again how they handle ties.  But Kendall also developed the W statistic for use in reliability analysis.  But that's where my knowledge of Kendall's W ends--I do not have a good reference book on the shelf to tap so I don't really know how W differes from tau... and where d fits into the mix.



-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Monday, December 13, 2010 2:13 PM
To: 'statalist@hsphsun2.harvard.edu'
Subject: st: RE: reliability with ordinal data-Kendall's w?

It seems to me that Somers' [NB] d is very well supported in Stata given Roger Newson's routines. 

I can't see much point to plots of the form re-invented by Bland and Altman in this case. Nor does the limits of agreement approach transfer other than queasily to a 4-point graded scale. -tabplot- from SSC offers one of various graphical alternatives. Diagonal agreement and off-diagonal disagreement will be pretty clear. 

Nick 
n.j.cox@durham.ac.uk 

Ploutz-Snyder, Robert (JSC-SK)[USRA]

I need to perform a reliability analysis among two raters who rated the same observations independently.  Observations are rated on an ordinal scale, taking 4 distinct values in increasing order.  Neither rater is assumed the "gold standard."  There are a LOT of ties in my dataset (a good thing in terms of rater reliability).

I came across some web hits about Kendall's W, which is different from the more familiar tau.  Can anyone attest to whether this is the best way to go about a reliability analysis with ordinal data?   Is there a way to implement on Stata?? How does W differ from tau, and Somer's d??  

Are there other/better methods that I should be considering too?

Also--for those knowledgeable about reliability analyses... I think that eventually I would like to graph these data with something akin to the Bland-Altman plots.  Do you take issue with that approach given that the data are not continuous?  Are the BA Limits of Agreement valid in this situation??


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/