Austin,
Thanks very much for the guidance against going to CIs, it's very
helpful (especially the note on proportions). I thought the thread
connection was Nick's question about why focus on comparing proportions
between units of government. But no problem with your conversion of the
subject line.
Thanks again.
arnold
Arnold H. Levinson, Ph.D.
Director, Tobacco Program Evaluation Group (TPEG)
Assistant Professor of Preventive Medicine
University of Colorado at Denver & Health Sciences Center
AMC Cancer Research Center
1600 Pierce Street, Lakewood CO 80214
[email protected]
303-239-3402
-----Original Message-----
From: Austin Nichols [mailto:[email protected]]
Sent: Wednesday, December 06, 2006 10:59 AM
To: [email protected]
Cc: Levinson, Arnold
Subject: conf int vs hypoth test [was: statistical test to compare ...]
Arnold.Levinson--
This is a very different question from the preceding thread's topic,
and should be asked with a different subject header, and of a
different listserv audience. That said, I don't think using
non-overlapping confidence intervals instead of hypothesis tests to
judge significance is a good idea, and will certainly give you wrong
answers in a regression framework (where the off-diagonal elements of
the covariance matrix play a role, ignored in confidence intervals) or
comparisons of proportions (confidence intervals around proportions
are notoriously problematic--see e.g. "Interval Estimation for a
Binomial Proportion" Lawrence D. Brown; T. Tony Cai; Anirban DasGupta
http://links.jstor.org/sici?sici=0883-4237%28200105%2916%3A2%3C101%3AIEF
ABP%3E2.0.CO%3B2-U).
Hard to say more without more detail about proposed comparisons, but
this comment should be more than enough...
On 12/6/06, [email protected] <[email protected]> wrote:
> Nick Cox said:
>
> "I'm still puzzled by the over-arching question and curious
> as to how this fits into any research project. ... Why is the state
mean
> vs other states mean comparison the focus here? ... is this
> really what other researchers want to know, given the
> many other possibilities such as confidence intervals,
> graphs, tables?"
>
> My comments are well beyond a Stata question, so if this is the wrong
> forum, please forgive and ignore what follows.
>
> We provide the state of Colorado with evaluation reports regarding
> programs to prevent and reduce tobacco use. Our audience isn't
> researchers but public and government data consumers. Sometimes we
> analyze population-level data collected at the U.S. national level
> through state-stratified designs, which support calculation of both
> state and national estimates. Other times, we analyze Colorado-level
> data collected under complex-sample designs that allow for estimation
at
> sub-state levels.
>
> In most cases, we are asked to present results comparatively, i.e.,
> between state and rest of nation, or sub-state area and rest-of-state.
> We typically declare (non)significance using design-adjusted
hypothesis
> tests of two rates, although we don't necessarily include p-values. So
> we do indeed focus on comparing means (proportions) by
> single-state-of-interest vs. rest-of-states, because it's exactly what
> we are asked to do (but not by other researchers).
>
> A second but related issue: We're thinking of dropping
> hypothesis-testing and relying instead on non-overlapping confidence
> bounds as the criterion for statistical difference. I think (but am
not
> sure) that this change would be neutral or conservative in all
instances
> (i.e., that non-overlap is always at least as stringent as appropriate
> hypothesis tests).
>
> My questions:
> 1. Does this context for focusing on mean comparisons by
> state-vs-other-states seem reasonable?
> 2. Are non-overlapping confidence intervals always more conservative
> than appropriate hypothesis tests?
> 3. What are the implications of switching to CIs for
> difference-determination (other than potential "customer"
disappointment
> that fewer comparisons appear significantly different)?
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
