What's this about?

Treatment-effects models extract experimental-style causal effects from observational data.

In experimental data, treatment groups must be assigned randomly, meaning characteristics across groups will be approximately equal.

Treatment-effects estimators reweight the observational data in hopes of achieving experimental-like balanced data results.

If the reweighting is successful, then the weighted distribution of each covariate should be the same across treatment groups. In such cases, we say that the treatment model "balanced" the covariates.

We can examine whether the treatment model balanced the covariates and perform a statistical test.

Three diagnostics and one test are provided.

1. One diagnostic reports, for each covariate, the model-adjusted difference in means in the treatment groups and the ratio of variances.

2. Another diagnostic graphs the model-adjusted estimated pdfs of covariates; these pdfs can be examined visually to verify that they are approximately equal.

3. The third graphical diagnostic is the same as the second but uses box plots rather than smoothed pdfs.

4. The statistical test is an overidentification test. It tests whether the model-adjusted means of the covariates are the same between groups.

Let's see it work

Say that we estimate the effect of smoking during pregnancy on infant birthweight using an inverse-probability-weighted (IPW) treatment-effects estimator.

We assume that treatment (smoking during pregnancy) is determined by marital status, the mother's age, attendance to prenatal care during the first quarter of pregnancy, and whether this is the mother's first pregnancy.

. teffects ipw (bweight) (mbsmoke mmarried mage prenatal1 fbaby 
                          c.mage#(c.mage i.mmarried prenatal1))

Iteration 0:   EE criterion =  9.365e-20
Iteration 1:   EE criterion =  3.141e-26

Treatment-effects estimation                    Number of obs     =      4,642
Estimator      : inverse-probability weights
Outcome model  : weighted mean
Treatment model: logit

                       
 
               Robust
               bweight 
      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]

ATE                    

               mbsmoke 

(smoker vs nonsmoker)  
  -239.6875   26.43427    -9.07   0.000    -291.4977   -187.8773

POmean                 

               mbsmoke 

            nonsmoker  
   3403.638    9.56792   355.73   0.000     3384.885     3422.39

We find that the average treatment effect (ATE) is -240 grams.

Have we done an adequate job of balancing the covariates so that we can trust the estimated treatment effect?

. tebalance overid, nolog

Overidentification test for covariate balance
         H0: Covariates are balanced:

         chi2(8)      =  11.8612
         Prob > chi2  =   0.1575

We cannot reject the null hypothesis that the covariates are balanced, and that's good.

We can look at the various diagnostics (and in real life, we probably would have used the diagnostics before using the statistical test).

tebalance summarize reports the model-adjusted difference in means and ratio of variances between the treated and untreated for each covariate:

. tebalance summarize

  Covariate balance summary
                                                   Raw     Weighted
                          
 
                          
Number of obs =        4,642      4,642.0
                          
Treated obs   =          864      2,329.1
                          
Control obs   =        3,778      2,312.9
                          




                  
 
Standardized differences          Variance ratio
                  
        Raw    Weighted           Raw   Weighted

         mmarried 
  -.5953009    .0053497      1.335944   .9953184
             mage 
   -.300179    .0410889      .8818025   1.076571
        prenatal1 
  -.3242695    .0009807      1.496155   .9985165
            fbaby 
  -.1663271   -.0130638      .9430944   .9965406
                  

             mage#

             mage 
  -.3028275    .0477465      .8274389   1.109134
                  

         mmarried#

             mage 

         married  
  -.6329701    .0197209      1.157026   1.034108
                  

        prenatal1#

             mage 

             Yes  
  -.4053969    .0182109      1.226363   1.032561

Ignore the raw columns, at least to begin, and focus on the weighted columns. Differences in weighted means are negligible, and variance ratios are all near one. The Raw columns show where we started, and, before weighting, differences were large.

tebalance can show us pdfs or box plots so that we can examine the entire distribution. Below we have put the graphs produced by tebalance density and tebalance box together:

Tests and diagnostics confirm that our model balances the covariates.

Tell me more

To find out more about checking for balance after teffects or stteffects, see [TE] tebalance.

Read the overview from the Stata News.