RE: st: Interrater agreement: finding the problematic items Date

["Ilian, Henry (ACS)" <Henry.Ilian@dfa.state.ny.us>]

Thanks Nick. That's what I'll do first thing Monday morning.

Henry

-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Friday, June 14, 2013 7:52 PM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: Interrater agreement: finding the problematic items Date

Install the specific package -ordvar- by

. ssc inst ordvar

and then read the help.

Nick
njcoxstata@gmail.com


On 14 June 2013 22:07, Ilian, Henry (ACS) <Henry.Ilian@dfa.state.ny.us> wrote:
> Thanks Mike,
>
> I clicked on the link, and there are a set of what appear to be modules with code.  I'm not sure how to use them. Could you give me a little more guidance?
>
> Henry
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of
> Lacy,Michael
> Sent: Friday, June 14, 2013 4:58 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: Interrater agreement: finding the problematic items
> Date
>
> Ilian, Henry (ACS)" <Henry.Ilian@dfa.state.ny.us> wrote:
>
>>I'm doing an interrater agreement study on a case-reading instrument. There are five reviewers using an instrument with 120 items.
>>The ratings scales are ordinal with either two, three or four options.
>>I'm less interested in reviewer tendencies than I am in problematic items, those with high levels of disagreement.
>>
>>Most of the interrater agreement/interrater reliability statistics
>>look at reviewer tendencies. I can see two ways of getting at
>>agreement on items. The first is to sum all the differences between
>>all possible pairs of reviewers, and those with the highest totals are the ones to examine. The other is Chronbach's alpha. Is there any strong argument for or against either approach, and is there a different approach that would be better than these?
>>
>
> My package -ordvar- (self-promotion mode on) might be of some use to you here.
> In your case, it would provide a 0/1 measure of the dispersion of each item's
> rating distribution.   Although not implemented in that package, an essentially
> identical measure (IOV) that may be better suited to data with small frequencies of raters is also cited in the help for that package.
>
>
> package ordvar from http://fmwww.bc.edu/RePEc/bocode/o
> ----------------------------------------------------------------------
> -----------------------------------------------------
>
> TITLE
>       'ORDVAR': module to calculate measures of ordinal consensus and
> dispersion
>
> DESCRIPTION/AUTHOR(S)
>
>       ordvar calculates measures of ordinal consensus  and dispersion.
>       These include lsq and 1-lsq, which are 0/1 normed ordinal
>       consensus and  dispersion statistics described in Blair and Lacy
>       (2000). [detailed citation
>
>  Blair, J. and M. Lacy. 2000. "Statistics of Ordinal Variation."
> Sociological Methods and Research. 28: 251-280
>
>  Berry, K. J. and P. W. Mielke, Jr. 1992. "Assessment of Variation in Ordinal Data." Perceptual and Motor Skills
>          74:63-66.
> ==============================================
>
>
> I also would suggest looking at Richard William's -oglm-.  It
> estimates location-scale ordinal response models, However, I don't
> know how well estimates from a regression model like that would
> perform with only
> 5 reviewers.
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/


Confidentiality Notice: This e-mail communication, and any attachments, contains confidential and privileged information for the exclusive use of the recipient(s) named above. If you are not an intended recipient, or the employee or agent responsible to deliver it to an intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and delete this communication from your computer. Thank you.


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/