I was led to believe that, when the assumptions for ANOVA hold, the power
and test size of the two approaches are practically indistinguishable
between them. The Monte Carlo simulation below seems to confirm this.
Looking at the three epsilons for the -anova , repeated()- of the
drugs-and-scores dataset, I inferred (mistakenly, it seems) that the
sphericity assumption wasn't grossly violated, and that is why I found the
large discrepancy between the two approaches so surprising.
Again, -simulate- (using the parameter estimates observed for that dataset)
could have told me how surprised I ought to have been about the magnitude of
the discrepancy, i.e., how rarely it will happen that there is such a large
discrepancy between the pooled-error approach and the separate t-test
approach when the assumptions of the repeated-measures ANOVA are met. In
the one-way ANOVA simulation below, a discrepancy as large or larger than
that observed for the drugs-and-scores dataset happens less than 0.5% of the
time.
-mtest()- does offer less conservative adjustments than the Bonferroni. See
also Roger Newson's -multproc- / -smileplot-.
Joseph Coveney
----------------------------------------------------------------------------
Richard Williams wrote:
At 09:31 AM 1/6/2004 -0600, David Airey wrote:
>>Joseph Coveney said,
>>
>>>I've tried to compare both approaches in the do-file below using the
dataset
>>>that Alan cited. The -mtest(bonferroni)- approach yields far more null
>>>hypothesis rejections than the corresponding Bonferroni-adjusted separate
>>>t-test approach. Such a discrepancy is surprising.
>>
>>This happens in a oneway between-subjects design too. The reason the
>>contrast is more powerful than the independent ttest is that the contrast
>>uses information for MSw from all groups and is therefore more accurate.
>>Of course this is true only if the homogeneity of variances assumption
>>holds. Perhaps along the same vein, the contrasts reject the null more
>>than the dependent ttests because more information is used for the
>>estimate of error, and in this case homogeneity of differences variances
>>holds? The dependent ttest is sometimes preferred because you can use
>>information from only those means tested, which is useful when sphericity
>>is violated.
>
>I just tried this with a oneway problem, and get the same kinds of results:
>The -mtest(bonferroni)- approach yields more null hypothesis rejections
>than the corresponding Bonferroni-adjusted separate t-test approach. These
>adjustments are already awfully conservative, and choosing an approach that
>makes them even more so is not very appealing! Although I guess you have
>to do what is right. I would think, though, that you would avoid the
>separate t-test approach unless it is clearly necessary, e.g. only go this
>route if homogeneity of variances does not hold?
----------------------------------------------------------------------------
clear
tempfile tmp
set more off
set seed 20040107
set obs 48
generate float score = .
generate byte trt = mod(_n, 4) + 1
save `tmp'
matrix trt12 = (0, -1, 1, 0, 0)
matrix trt13 = (0, -1, 0, 1, 0)
matrix trt14 = (0, -1, 0, 0, 1)
matrix trt23 = (0, 0, -1, 1, 0)
matrix trt24 = (0, 0, -1, 0, 1)
matrix trt34 = (0, 0, 0, -1, 1)
matrix trts = trt12 \ trt13 \ trt14 \ trt23 \ trt24 \ trt34
matrix drop trt12 trt13 trt14 trt23 trt24 trt34
*
program define ansim, rclass
syntax , [Hypothesis(integer 0)]
replace score = invnorm(uniform()) + `hypothesis' * trt / 2
anova score trt
test , test(trts) mtest(noadjust)
matrix A = r(mtest)
forvalues i = 1/6 {
return scalar pmse`i' = A[`i', 3] < 0.05
}
forvalues i = 1/3 {
local j = `i' + 1
forvalues k = `j'/4 {
ttest score if trt == `i' | trt == `k', by(trt)
return scalar stts`i'`k' = r(p) < 0.05
}
}
end
*
forvalues Ha_is_true = 0/1 {
quietly simulate "ansim , h(`Ha_is_true')" ///
pmse1 = r(pmse1) pmse2 = r(pmse2) ///
pmse3 = r(pmse3) pmse4 = r(pmse4) pmse5 = r(pmse5) ///
pmse6 = r(pmse6) stts1 = r(stts12) stts2 = r(stts13) ///
stts3 = r(stts14) stts4 = r(stts23) stts5 = r(stts24) ///
stts6 = r(stts34), reps(3000)
generate int rec = _n
quietly reshape long pmse stts, i(rec) j(trt)
label variable pmse "Pooled error"
label variable stts "Separate error"
display
display as text "Ha " as result `Ha_is_true'
summarize pmse stts
* fsum pmse stts, stats(mean) format(%9.3f) label
tabulate pmse stts, taub
display
if `Ha_is_true' == 0 {
quietly use `tmp', clear
}
}
* How surprised do I have a right to be for, say, a
* discrepancy at least as great as 2 rejections
* versus 4 rejections out of 6 comparisons given these
* parameters?
foreach var of varlist pmse stts {
bysort rec: generate byte tnp`var' = sum(`var')
}
by rec: keep if _n == _N
generate byte surprised = (tnpstts <= 2) & ///
(tnppmse >= 4)
summarize surprised
exit
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
