st: Cohen's Kappa or other interrater agreement statistics

["Ilian, Henry (ACS)" <Henry.Ilian@dfa.state.ny.us>]

I have a case reading instrument with 120 items. Most of the items have three answer choices, but some have two and some have four. There is also one with five. We did an inter-rater reliability study with five readers reading the same case. I haven't looked at the data yet, but I'll need to soon. I was going to use Cohen's Kappa, which is available in SPSS. I was going to set up the data file so that similar items--related topic areas with the same number of answer choices--were treated as separate cases, analogous to a set of MRIs, Xrays, etc. rated by different clinicians, as I've seen in a number of articles. I'm not entirely happy about this approach because the content for the items may not be similar enough. Also, it doesn't allow me to identify items that are troublesome.

Can anyone see a better way of testing for interrater agreement with this type of data?

Thanks,

Henry


Confidentiality Notice: This e-mail communication, and any attachments, contains confidential and privileged information for the exclusive use of the recipient(s) named above. If you are not an intended recipient, or the employee or agent responsible to deliver it to an intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and delete this communication from your computer. Thank you.


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/