Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: st: RE: P-values for the difference in sensitivity in metandi
From
Joe Canner <[email protected]>
To
"[email protected]" <[email protected]>
Subject
RE: st: RE: P-values for the difference in sensitivity in metandi
Date
Tue, 25 Mar 2014 20:29:29 +0000
Carole,
"Statistically significant" implies a hypothesis and a statistical test to evaluate that hypothesis. What is the hypothesis in this case? What is being compared? As you correctly noted, comparing a sensitivity/specificity to 0% is trivial and non-sensical. Instead you need to come up with a hypothesis that makes sense clinically. Do you want patient-collected samples to be at least 80% as good as physician-collected? 90%? What is the impact of false negatives and false positives? This is not a statistical question, per se, but perhaps if you provide more background we can guide in considering the important factors that one might use in constructing an appropriate hypothesis.
Regards,
Joe
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Carole Lunny
Sent: Tuesday, March 25, 2014 3:38 PM
To: [email protected]
Subject: Re: st: RE: P-values for the difference in sensitivity in metandi
Dear Joe,
I am comparing self-collected samples compared to the gold standard
clinician-collected samples. The confidence intervals for absolute
sensivity and specificity will always be positive and withing the
range 0.1 - 1.0, so this is not a way to tell is it is statistically
significant.
Regards, Carole
On Tue, Mar 25, 2014 at 12:11 PM, Joe Canner <[email protected]> wrote:
> Carole,
>
> I don't see anywhere in your data where you distinguish between physician-collected and patient-collected samples. Is there a gold standard that you are comparing to? Or is physician-collected the gold standard?
>
> For the time being I am assuming the latter, in which case there is no p-value to calculate. You are calculating a single sensitivity (0.825) and a single specificity (0.924), so there is nothing to compare. Evaluating sensitivity and specificity is mostly a matter of clinical judgment, e.g., which is more problematic false negatives or false positives? What is the prevalence of the disease in question? This combination of sensitivity and specificity look pretty good, but depending on the clinical context they may be totally inadequate.
>
> If you are talking about the former (comparing physician-collected and patient-collected with some gold standard), I'm not aware of such a thing (but don't take that to mean much; I didn't even know about -metandi- until you mentioned it and I could really have used it a few months ago). However, a very legitimate (and to some, preferable) way to do that comparison is to see if the 95% confidence intervals overlap. More to the point, however, is that the data you presented doesn't seem to represent a comparison between two methods versus a gold standard.
>
> Regards,
> Joe
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of Carole Lunny
> Sent: Tuesday, March 25, 2014 2:43 PM
> To: [email protected]
> Subject: Re: st: RE: P-values for the difference in sensitivity in metandi
>
> Dear Joe,
>
> Thanks for replying. We are comparing the accuracy of self-collected
> samples compared to clinician-collected samples.
>
> Here is the data:
>
> tp fp tn fn
> 274 55 89 1510
> 238 101 74 2223
> 57 6 10 439
> 53 5 7 26
> 48 10 13 64
> 144 0 11 1039
> 6 3 1 42
> 53 9 9 85
> 75 10 9 41
> 31 2 4 11
> 50 22 19 285
> 9 7 4 73
> 28 4 8 63
> 85 32 9 121
> 67 24 17 114
> 31 7 10 51
> 49 5 2 258
> 17 3 0 25
> 30 0 12 12
> 62 13 0 71
> 407 59 23 1899
> 72 34 40 173
> 86 6 2 24
> 63 51 28 594
> 9 4 10 162
> 123 54 37 704
> 157 31 18 923
> 170 128 132 985
>
> From here you get the sensitivity of self-collected samples compared
> to clinician-collected sample (see my results). I am looking for the
> p-value of the difference between the sensitivity of the
> self-collected samples compared to the clinician-collected samples.
> (and specificity too).
>
> Thanks for your help, Carole
>
>
> On Tue, Mar 25, 2014 at 11:35 AM, Joe Canner <[email protected]> wrote:
>> Carole,
>>
>> You say you want the "difference in sensitivity/specificity". What are you calculating the difference *between*? In your -metandi- command all I see is the components for computing a single sensitivity and specificity, not for comparing the sensitivity/specificity between two different modalities or two different thresholds.
>>
>> Regards,
>> Joe Canner
>> Johns Hopkins University School of Medicine
>>
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On Behalf Of Carole Lunny
>> Sent: Tuesday, March 25, 2014 2:02 PM
>> To: [email protected]
>> Subject: st: P-values for the difference in sensitivity in metandi
>>
>> Dear Stata users,
>>
>> I am using metandi to get results of a diagnostic accuracy review I am
>> doing. I am using the HSROC model, and would like to know how to get
>> the P-values for the difference in sensitivity/specificity in metandi
>> (or another stata command).
>>
>> These are the results from metandi:
>>
>> . metandi tp fp fn tn
>>
>> Refining starting values:
>>
>> Iteration 0: log likelihood = -233.08848
>> Iteration 1: log likelihood = -230.8551
>> Iteration 2: log likelihood = -230.79615
>> Iteration 3: log likelihood = -230.74257
>>
>> Performing gradient-based optimization:
>>
>> Iteration 0: log likelihood = -230.74257
>> Iteration 1: log likelihood = -230.74239
>> Iteration 2: log likelihood = -230.74239
>>
>> Meta-analysis of diagnostic accuracy
>>
>> Log likelihood = -230.74239 Number of studies = 31
>> ------------------------------------------------------------------------------
>> | Coef. Std. Err. z P>|z| [95% Conf. Interval]
>> -------------+----------------------------------------------------------------
>> Bivariate |
>> E(logitSe) | 1.552802 .1604973 1.238233 1.867371
>> E(logitSp) | 2.490983 .1836023 2.131129 2.850837
>> Var(logitSe) | .6461271 .2113009 .3403707 1.226546
>> Var(logitSp) | .8975503 .2730492 .4944354 1.629326
>> Corr(logits) | .1192086 .209656 -.2886231 .4904251
>> -------------+----------------------------------------------------------------
>> HSROC |
>> Lambda | 3.980269 .2660604 3.4588 4.501737
>> Theta | -.3043524 .2406498 -.7760173 .1673124
>> beta | .1643365 .2221673 0.74 0.459 -.2711034 .5997763
>> s2alpha | 1.704627 .5034386 .9555173 3.041027
>> s2theta | .3353756 .1085868 .1778002 .632602
>> -------------+----------------------------------------------------------------
>> Summary pt. |
>> Se | .8253181 .0231386 .7752564 .8661538
>> Sp | .9235073 .01297 .8938922 .9453619
>> DOR | 57.04186 14.58478 34.55841 94.15288
>> LR+ | 10.7895 1.883955 7.662542 15.19251
>> LR- | .1891505 .0254589 .1452912 .2462498
>> 1/LR- | 5.286795 .7115817 4.060917 6.882731
>> ------------------------------------------------------------------------------
>> Covariance between estimates of E(logitSe) & E(logitSp) .0029529
>>
>> Any help would be appreciated. Thank you, Carole
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>> * http://www.ats.ucla.edu/stat/stata/
>>
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>> * http://www.ats.ucla.edu/stat/stata/
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/faqs/resources/statalist-faq/
> * http://www.ats.ucla.edu/stat/stata/
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/faqs/resources/statalist-faq/
> * http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/