RE: st: Comparing risk scores

["Jesper Lindhardsen" <JESLIN01@geh.regionh.dk>]

Hi there,

A popular, although perhaps rough, method is to use logist models and determine the best models by AUC. However in my view , this seem to impose restrictions on the data you analyse regarding censoring and equal follow-up. Since (I gather) your data you use to make the risk scores is survival data, perhaps the approach by Newson could be applied (Newson RB. Comparing the predictive power of survival models using Harrell ' s c or Somers ' D. Stata Journal - http://www.imperial.ac.uk/nhli/r.newson/papers/predsurv.pdf), where comparison of Harrells C of Cox regression is used to compare the prediction of the scheme.

Incidentially, I guess generalisability is always an issue and the risk scores should be tried out in other cohorts than the one you use to develop the model.

HTH

Jesper

Jesper Lindhardsen
MD, PhD candidate
Department of Cardiovascular Research
Copenhagen University Hospital, Gentofte
Denmark






-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of K Jensen
Sent: 18 October 2011 14:36
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: Comparing risk scores

The clinicians I am working with are ADAMANT that they want a simple
scale based on ticking boxes and adding the number of ticks, and
nothing more complicated...

If we work within these constraints, how is it best to compare the
possible scores?

Thanks

Karin

On 18 October 2011 13:22, Richard Goldstein <richgold@ix.netcom.com> wrote:
> Karin,
>
> I suggest you might want to read Sullivan, LM, et al. (2004),
> "Presentation of multivariate data for clinical use: the Framingham
> study risk score functions," _Statistics in Medicine_, 23: 1631-1660,
> which describes how the Framingham people came up with their risk scores
>
> Rich
>
> On 10/18/11 8:18 AM, K Jensen wrote:
>> Hi Nick
>>
>> Thanks for your reply.  It's actually a bit more complicated than
>> that.  We are trying to construct a "best" single score that would be
>> simple and used clinically.  The elements that are summed to make the
>> score (0,1,2,3 etc) are derived from various clinical measurements.
>> They are dichotomised by choosing the cutpoint that maximises the sum
>> of sensitivity+specificity.  Only those binary variables significant
>> in a univariate logistic regression are proposed for the model.
>>
>> I am wanting to choose the "best" model, that is useful for
>> clinicians.  If we had 7 binary variables, say, I would look at all
>> possibilities of choosing different combinations of the sums of them.
>> E.g. 1, 2, 3, 4, 5, 6, 7,1+2,1+3,1+4,1+5,1+6,1+7, 2+3, 2+4,... up to
>> 1+2+3+4+5+6+7.  I would like to use the optimal score based on this
>> method, but don't know how to measure optimality.
>>
>> Best wishes,
>>
>> Karin
>>
>> On 18 October 2011 12:36, Nick Cox <njcoxstata@gmail.com> wrote:
>>> I would recast this as a -logit- or -logistic- problem in which your
>>> outcome is dead or alive. Depending on how you think about your
>>> scores, they define predictors to be treated as they come or
>>> predictors to be treated as a set of indicator variables (or in some
>>> cases both).
>>>
>>>  I don't think you are restricted to using one score or the other as predictor.
>>>
>>> Nick
>>>
>>> On Tue, Oct 18, 2011 at 12:11 PM, K Jensen <k.x.jensen@gmail.com> wrote:
>>>> Maybe this is more of a stats question than a Stata one, but there are
>>>> such a lot of good brains here...
>>>>
>>>> We are constructing point scores to indicate severity of risk  Death
>>>> is the outcome. What is the best way of measuring the usefulness of
>>>> the score?  The aim is to show a good gradient of risk.  Say the
>>>> results for two different scores were:
>>>>
>>>> Score  Dead  Alive    %dead    Totals
>>>> 0        12    136      9.9%      145
>>>> 1        18    126     15.4%      144
>>>> 2        18     62     26.2%       81
>>>> 3        10      9     57.1%       20
>>>> 4         2      0    100  %        3
>>>> -------------------------------------
>>>> Total:   60    333                393
>>>>
>>>> Score  Dead  Alive    %dead    Totals
>>>> 0         8    174      4.6%      182
>>>> 1        21    143     12.8%      164
>>>> 2        22     19     53.7%       41
>>>> 3         5      1     83.3%        6
>>>> -------------------------------------
>>>> TOTAL:   60    333                393
>>>>
>>>> Which is the better score?  What is the best way to measure its
>>>> predictive power?  I understand that ROC type analysis doesn't really
>>>> apply here.  Some measure of R-squared?  AIC?
>>>>
>>>> Thankyou
>>>>
>>>> Karin
>>>>
>>>> PS) I have made up the data, so the numbers don't quite add up.  It is
>>>> meant to be two different, competing scores on the same people.
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/