Re: st: Area under the ROC curve less than 0.5?

[Roger Newson <r.newson@imperial.ac.uk>]

In reply to your first query, the area under the ROC can be anything 
from 0 to 1. Predictors with a ROC area less than 0.5 are negative 
prwedictors, and predictors with a ROC area between 0.5 and 1 are 
positive predictors.

In reply to your second query, it is possible for the difference between 
2 ROC areas to be statistically non-significant in a small sample, even 
if one ROC area is less than half the other. This is because the 
confidence interval for the difference between 2 ROC areas may be wide, 
and include a zero difference and a range of positive and negative 
differences. To get a confidence interval for the difference between 2 
ROC areas (also known as Harrell's c statistics), use the -somersd- 
package, which you can download from SSC and which has 3 .pdf manuals, 
distributed with the package as ancillary files. The on-line help 
contains hyper-references to the Stata Journal articles explaining the 
-somersd- package, which you can use with -lincom- to get confidence 
intervals for the difference between 2 Harrell's c statistics.

I hope this helps.

Best wishes

Roger


Roger B Newson BSc MSc DPhil
Lecturer in Medical Statistics
Respiratory Epidemiology and Public Health Group
National Heart and Lung Institute
Imperial College London
Royal Brompton Campus
Room 33, Emmanuel Kaye Building
1B Manresa Road
London SW3 6LR
UNITED KINGDOM
Tel: +44 (0)20 7352 8121 ext 3381
Fax: +44 (0)20 7351 8322
Email: r.newson@imperial.ac.uk
Web page: http://www.imperial.ac.uk/nhli/r.newson/
Departmental Web page:
http://www1.imperial.ac.uk/medicine/about/divisions/nhli/respiration/popgenetics/reph/

Opinions expressed are those of the author, not of the institution.

On 21/01/2011 09:56, reggae.benigno@metrobank.com.ph wrote:
> Hi!
>
> I just want to ask, after running a logistic regression on a development
> sample, I ran the lroc command. The corresponding area under the ROC
> curve is .9226. When I wished to check the robustness of the model on a
> holdout sample, I keyed in "lroc if sample==2". The software was able to
> run it on this separate sample, however, the resulting AROC is now
> .4060. I thought AROC values are only from 0.5 to 1.0, or was I
> mistaken?
>
> Also, when I tried running roccomp on the two samples, Stata was again
> able to churn out the same ROCs (.9226 and .4060), but the chi-square
> p-value is .0663, indicating that at alpha=.05, the null hypothesis of
> equal areas under the curves cannot be rejected.  Does it make sense
> that even though the AROC of the second sample is less than half of the
> first sample, the test still says that the 2 ROC curves have equal
> areas? Is this test perhaps affected by the number of observations?
> (sample 1 : 26 positive, 174 negative, while sample 2 contains 5(+) and
> 102(-)).
>
> Would really appreciate anyone's help. Thanks in advance!
>
>
> - Reggae
>
>
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/