How do you compare? Examining effects in multilevel models using contrasts
If your statistical background includes ANOVA modeling and analysis of designed experiments, you are likely already familiar with contrasts. If not, contrasts may be a new concept, or perhaps you use certain types of contrasts but call them by different names. So before we jump to contrasts with multilevel models, let’s look at a few types of tests you can perform using Stata’s contrast command.
Making comparisons using contrast
As with the example data in Keppel and Wickens (2004, chapter 13), let’s suppose we have fifth-grade students who are asked to learn a set of vocabulary words and are tested on them a week later. We are interested in comparing three teaching methods and four types of words. When teaching, the teacher provides no verbal feedback, positive feedback, or negative feedback. The four word lists are categorized by the frequency with which the words are used, where freq=1 corresponds to words least frequently used and freq=4 corresponds to words most frequently used.
Let's start by regressing vocabulary test score on the frequency categories.
The coefficients in the regression output are one type of contrast—comparisons with the base level of least frequently used words. Mean test scores for the third and fourth frequency levels are statistically greater than those for the first (base) level. However, it would be nice to know whether differences exist for each increase in word frequency level. Are there differences between the second and third levels? Between the third and fourth levels? We can use the ar. contrast operator to make these “reverse adjacent” comparisons.
Here we find that the change from level 2 to level 3 is the only one with a statistically significant difference in estimated mean scores.
We can see this large jump when we plot the estimated means using margins freq followed by the marginsplot command.
Now, let’s fit a model allowing for an interaction between frequency and type of feedback by typing
. regress score freq##feedback (output omitted)
We can use contrast to obtain tests of the main and interaction effects.
We have statistical evidence for an interaction effect, but at this point, we can’t say much about where differences exist. A plot allows us to visualize the interaction.
We can test specific hypotheses about the relationship between word frequency and teaching method. For example, when positive feedback is given, feedback=2, does word frequency have an effect?
There is no statistically significant difference in the four means for this teaching method.
We can also test whether there is an effect of teaching method at individual frequency levels.
For the first and second frequency levels (but not the third and fourth), there is a statistical difference in the estimated means for the different teaching methods.
Is there really any interaction if we consider only the change from the first to second frequency level?
We do not find evidence of an interaction effect at these frequency levels.
Multilevel models
We can easily apply the same types of contrasts when fitting multilevel models. For example, suppose that each student was tested multiple times. We can fit a random-effects model allowing for student-level variation in the intercepts and perform the same types of contrasts. To do this, we could type
. mixed score feedback##freq || id: . contrast feedback##freq . contrast feedback@frequency
However, acknowledging that we have repeated measurements on students and a fairly small sample size, we request that both mixed and contrast report small-sample tests using a repeated-measures ANOVA method for computing denominator degrees of freedom.
As before, we find an interaction between the type of feedback and the frequency of word use. Using the contrast command below, we also find that the simple effects of feedback exist only for the lower-frequency words.
And beyond
We have considered fairly traditional experimental design applications. However, the use of contrast is not so limited. contrast can be used after fitting most models in Stata. Options are available for working with unequal group sizes and for adjusting results for multiple comparisons. Find details on these and many other extensions in [R] contrast.
—Kristin MacDonald
Asst. Director of Statistical Services
Reference
Keppel, G., and T. D. Wickens. 2004. Design and Analysis: A Researcher’s Handbook. 4th ed. Upper Saddle River, NJ: Pearson.
