Re: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science

[Nick Cox <njcoxstata@gmail.com>]

Thanks for the extra context. We can all agree that what you should be
doing and how to do it in Stata [NB] are different questions. We're
not examiners or reviewers, but my guess is that the same kind of
questions would recur if this were part of a thesis, report or
submitted paper.

Nick

On Fri, Mar 1, 2013 at 2:37 AM, Lenny Lesser <lenny3200@gmail.com> wrote:

> Thank you all for your comments.
> Surely their is a methods and conceptual issue.
> There is also a STATA question, which I was mostly discussing here.
>
> The problem with our scale is that is wasn't meant for smartphone
> apps.  We actually developed another shorter scale for this purpose
> that we tested along side this.  The shorter scale had more agreement
> between raters and ranked the apps similarly.  The reason the first
> scale was so poor is that it asked detailed questions that were easy
> to miss in looking at a smartphone app.  They were also repetitive and
> similar.
>
> For the STATA question, it appears that we've found the right solution.

> On Thu, Feb 28, 2013 at 5:37 AM, JVerkuilen (Gmail)

>> On Thu, Feb 28, 2013 at 6:10 AM, Nick Cox <njcoxstata@gmail.com> wrote:
>>> I do agree broadly with Allan, whether or not that is surprising.
>>
>> Me too, actually, though I think both Nick and I were misquoted and
>> taken out of context. I know the first thing I did with the raw data
>> was qnorm and graph box....
>>
>>
>>> A wilder idea is that rater 4 who gave no score higher than 3 either
>>> never knew or somehow forgot that scores could be up to 100 and just
>>> used a 5-point scale. Even if #4 did know that, #4 is so out-of-line
>>> that including them remains dubious, although doing computations with
>>> and without #4 remains manageable.
>>>
>>> In any case if the highest score is 18, then something else is going
>>> on that needs to be spelled out, if only as context.
>>
>> 100% agree and that's certainly consistent with a good bit of the
>> discussion on that thread, but there were quite a bit of discussions
>> not strictly aimed at Lenny's original problem but at the broader
>> question of ICC estimation using Stata, which is the nature of this
>> listserv.
>>
>>
>>
>>> Only the original poster can add more context than we already have. In
>>> any field that I know about this dataset would be too small to be
>>> publishable, except as a toy dataset to make points about method
>>> (which I take it is Jay's motive here).
>>
>> Yes, that's my exact motive. I'm writing a paper on "small sample"
>> problems with estimation of reliability coefficients, which are quite
>> common in practice.
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/