Arnold.Levinson--
This is a very different question from the preceding thread's topic,
and should be asked with a different subject header, and of a
different listserv audience. That said, I don't think using
non-overlapping confidence intervals instead of hypothesis tests to
judge significance is a good idea, and will certainly give you wrong
answers in a regression framework (where the off-diagonal elements of
the covariance matrix play a role, ignored in confidence intervals) or
comparisons of proportions (confidence intervals around proportions
are notoriously problematic--see e.g. "Interval Estimation for a
Binomial Proportion" Lawrence D. Brown; T. Tony Cai; Anirban DasGupta
http://links.jstor.org/sici?sici=0883-4237%28200105%2916%3A2%3C101%3AIEFABP%3E2.0.CO%3B2-U).
Hard to say more without more detail about proposed comparisons, but
this comment should be more than enough...
On 12/6/06, [email protected] <[email protected]> wrote:
Nick Cox said:
"I'm still puzzled by the over-arching question and curious
as to how this fits into any research project. ... Why is the state mean
vs other states mean comparison the focus here? ... is this
really what other researchers want to know, given the
many other possibilities such as confidence intervals,
graphs, tables?"
My comments are well beyond a Stata question, so if this is the wrong
forum, please forgive and ignore what follows.
We provide the state of Colorado with evaluation reports regarding
programs to prevent and reduce tobacco use. Our audience isn't
researchers but public and government data consumers. Sometimes we
analyze population-level data collected at the U.S. national level
through state-stratified designs, which support calculation of both
state and national estimates. Other times, we analyze Colorado-level
data collected under complex-sample designs that allow for estimation at
sub-state levels.
In most cases, we are asked to present results comparatively, i.e.,
between state and rest of nation, or sub-state area and rest-of-state.
We typically declare (non)significance using design-adjusted hypothesis
tests of two rates, although we don't necessarily include p-values. So
we do indeed focus on comparing means (proportions) by
single-state-of-interest vs. rest-of-states, because it's exactly what
we are asked to do (but not by other researchers).
A second but related issue: We're thinking of dropping
hypothesis-testing and relying instead on non-overlapping confidence
bounds as the criterion for statistical difference. I think (but am not
sure) that this change would be neutral or conservative in all instances
(i.e., that non-overlap is always at least as stringent as appropriate
hypothesis tests).
My questions:
1. Does this context for focusing on mean comparisons by
state-vs-other-states seem reasonable?
2. Are non-overlapping confidence intervals always more conservative
than appropriate hypothesis tests?
3. What are the implications of switching to CIs for
difference-determination (other than potential "customer" disappointment
that fewer comparisons appear significantly different)?
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
