RE: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science

["Allan Reese (Cefas)" <allan.reese@cefas.co.uk>]

Lenny Lesser posed a problem (copied below) on Statalist and has
received several replies to the question posed on ICC method.  Nick Cox
commented "A scatter plot matrix is instructive" and in a follow-up
message added, "it is well to know what patterns do or do not exist
before you start quantifying them.  This is just to underline that
repeating the graphs with ranks underlines how much information is
thereby discarded."  There have been various comments on methods and
Stata commands, including JVerkuilen's, "my gut impression is that you
really should use the scores. The context you cite is no doubt correct,
but for comparing raters with each other the scores they gave are
essential."

With no disrespect to anyone, I see this as a classic example of
"Mathsworld" - a mindset where because you are presented with numbers
you do sums, and the context is ignored.  LOOK AT THE DATA.

In the first place, the score values are not skewed; they are just all
low range.  The highest score is 18/100, adjustable to A* in GCSE but
indicating to me a general problem with the exercise.  The "outlier"
rater 4 has awarded values ONLY of 0/1/2/3, from which I'd guess she has
ranked the apps anyway and never awarded scores. 

. table Score Rater

----------------------------------
          |         Rater         
    Score |    1     2     3     4
----------+-----------------------
        0 |                      1
        1 |          1           8
        2 |    1     1     1     1
        3 |          2           1
        5 |    1     1     1      
        6 |    2           3      
        7 |    2     1     1      
        8 |          1            
        9 |          1            
       10 |          1            
       11 |    1           1      
       12 |          2            
       13 |    1           1      
       15 |                1      
       16 |    1           1      
       17 |    1           1      
       18 |    1                  
----------------------------------

Perhaps everyone noticed that, and has treated this as purely an
exercise in modelling?  However, it is my impression that people
sometimes love their models more than the data they "explain". 

Allan


------------------------------------------------------------------------
--------