Re: st: Re: cutoff point for ROC curve

[Harrison Alter <doctordad@gmail.com>]

That did the trick!  Thanks so much.  (Now I just have decide to
upgrade from v11.)

Thanks for your help,

Harrison Alter

On Wed, Oct 16, 2013 at 9:01 PM, Phil Clayton
<philclayton@internode.on.net> wrote:
> It's only just been added to SSC, this is probably a caching problem. Try:
> ssc install cutpt
>
> If that doesn't work email me off-list and I'll send it to you.
>
> Phil
>
> On 17/10/2013, at 2:52 PM, Harrison Alter <doctordad@gmail.com> wrote:
>
>> Hi Phil -
>> I tried
>>
>> findit cutpt
>>
>> because I have some diagnostic data I wanted to try it on, but
>> -findit- returned error 601.  Is -cutpt- still live?
>>
>> Thanks,
>> Harrison Alter
>>
>> On Wed, Oct 16, 2013 at 6:09 PM, Phil Clayton
>> <philclayton@internode.on.net> wrote:
>>> I agree with Clyde that there is more to picking a cutpoint than minimising errors. I also think it is not ideal for the cutpoint to be entirely data-driven, even if you have assigned weights to the different types of errors.
>>>
>>> Nevertheless sometimes this technique does have a place and a while ago I wrote a small program to estimate the "optimal" cutpoint using the Youden index, or the method of Liu et al, or the point closest to (0,1) on the ROC curve.
>>>
>>> Thanks to Kit Baum this package is now available on SSC as -cutpt-
>>>
>>> By default -cutpt- uses an adjustment suggested by Fluss, although this can be suppressed.
>>>
>>> I don't agree that the Youden or Liu methods are arbitrary. For a binary test the area under the ROC curve is a function of the sum of the sensitivity and the specificity, so maximising this sum (Youden method) maximises the ROC AUC. The concordance statistic is given by the product of sensitivity and specificity, so maximising this product (Liu method) optimises test discrimination.
>>>
>>> As far as I can tell the point closest to (0,1) on the ROC curve doesn't have any meaningful interpretation so I included this in -cutpt- only for completeness.
>>>
>>> Phil
>>>
>>> Fluss R, Faraggi D, Reiser B. Estimation of the Youden Index and its associated cutoff point. Biom J. 2005 Aug;47(4):458–72.
>>>
>>> Liu X. Classification accuracy and cut point selection. Stat Med. 2012 Oct 15;31(23):2676–86.
>>>
>>> On 15/10/2013, at 8:55 AM, Clyde Schechter <clyde.schechter@gmail.com> wrote:
>>>
>>>> I would advise Michael Stewart not to seek some arbitrary formula for
>>>> the optimal cut-off point.  He doesn't say what is being classified,
>>>> but regardless, the substantive issue is the trade-off between two
>>>> types of misclassification errors: false negatives and false
>>>> positives.  Both types of error have consequences, usually different.
>>>> To find an optimal cut-point requires assigning a loss to each type of
>>>> error and then expressing the expected loss in terms of sensitivity,
>>>> specificity and prevalence of the attribute being identified by the
>>>> classification.  Then you pick the cut-off which minimizes the
>>>> expected loss.
>>>>
>>>> My practical experience with this process is that people are often
>>>> reluctant to quantify the losses associated with each type of error,
>>>> because the losses are often of a qualitatively different nature.  For
>>>> example, a missed diagnosis may lead to loss of life, whereas a false
>>>> positive diagnosis may lead to unnecessary surgery.  How does one
>>>> assign values to those?  Not easily.
>>>>
>>>> So it feels more comfortable to seize on some simple formula, such as
>>>> the sum of sensitivity and specificity.  Nevertheless, if you don't
>>>> really quantify and compare the losses associated with each type of
>>>> error, applying some arbitrary formula will give you only the
>>>> illusion, not the reality, of optimality.  One is simply optimizing an
>>>> arbitrary quantity that bears no relation to the matter at hand.
>>>>
>>>> Clyde Schechter
>>>> Dept. of Family & Social Medicine
>>>> Albert Einstein College of Medicine
>>>> Bronx, New York, USA
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>
>>>
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>>
>>
>>
>> --
>> Harrison Alter, MD, MS, FACEP
>> Research Director
>> Department of Emergency Medicine
>> Alameda Health System - Highland Hospital
>> 1411 East 31st Street
>> Oakland, California 94602
>>
>> Executive Director
>> Andrew Levitt Center for Social Emergency Medicine
>> levittcenter.org
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/



-- 
Harrison Alter, MD, MS, FACEP
Research Director
Department of Emergency Medicine
Alameda Health System - Highland Hospital
1411 East 31st Street
Oakland, California 94602

Executive Director
Andrew Levitt Center for Social Emergency Medicine
levittcenter.org

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/