RE: Re: st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow?

["Ariel Linden, DrPH" <ariel.linden@gmail.com>]

Simone,

Adam gave you good advice and pointed you to a good article that discusses
why you shouldn't use t-tests for testing covariate balance. Rather than
re-asking the same question, perhaps you should read the article and learn
why t-tests are not a good measure of balance (hint: we are not making
inferences, and sample size can affect the results), whereas standardized
differences, variance ratios, and a whole host of graphic plots (q-q plots,
box plots, kernel density plots, etc.) are more suitable.

You will likely not be asking this question again, once you've done your
homework.

Additional suggested reading (a few among hundreds of articles):

Caliendo M, Kopeinig S. Some practical guidance for the implementation of
propensity score matching. Journal of Economic Surveys 2008;22:31-

Stuart EA. Matching methods for causal inference: a review and a look
forward. Statistical Science 2010;25(1):1-21.

Flury BK, Reidwyl H. Standard distance in univariate and multivariate
analysis. The American Statistician 1986;40:249-251

Rubin DB. Using propensity scores to help design observational studies:
Application to the tobacco litigation. Health Services & Outcomes Research
Methodology 2001;2:169-188.

Iacus SM, King G, Porro G. Multivariate matching methods that are monotonic
imbalance bounding. Journal of the American Statistical Association
2011;106:345-361.

Ariel
Date: Sun, 13 Jan 2013 12:21:06 +0100
From: simone ferro <simone.ferro88@gmail.com>
Subject: Re: st: problem with the interpretation of pstest after psmatch2,
t-tests and percentage of bias provide conflicting results, which one should
I follow?

Thank you so much for your answer,
it was very clear and helpful,
I followed your suggestion and I got better results, so thank you very much,
by the way I'm using a kernel matching based on a probit regression,
variables are mainly continuous.
Now I get a better balance, but as expected t-tests refuse the
null-hypotesis of equal means, and chi2 test refuse the null hypotesis of
balance as well, looking at the means it seems to me that this is a good
balance, do you think I can simply don't look at t-tests and chi2 test?
- ---------------------------------------------------------
             |       Mean               |     t-test
    Variable | Treated Control    %bias |    t    p>|t|
- -------------+--------------------------+----------------
cost_per_e~l |  60.95   60.053      4.4 |  -3.48  0.001
asset        | 193.35   260.33     -5.3 |  -1.41  0.159
employees    | 398.82   551.06     -7.9 |  -0.97  0.333
av_age       | 54.424   54.951     -6.9 |   1.69  0.092
donne_perc   | .29114   .28214      4.3 |   3.86  0.000
laureati_p~c | .23952   .23494      1.9 |   0.11  0.914
bluecollar~c | .35553   .37312     -5.8 |  -2.94  0.004
occupaz_min  | .16463   .13253      9.1 |  -0.13  0.895
donne_perc   | .29114   .28214      4.3 |   3.86  0.000
- ----------------------------------------------------------
Pseudo R2      LR chi2        p>chi2      MeanB     MedB
- ----------------------------------------------------------
    0.163        86.83         0.000       5.6       5.3
- ----------------------------------------------------------

thank you again,
Regards,
Simone

Il giorno 13/gen/2013, alle ore 05:18, Adam Olszewski
<adam.olszewski@gmail.com> ha scritto:

> Hi Simone,
> t-test based comparisons after PS matching are highly controversial.
> t-test makes a lot of usually untenable assumptions (are the variables 
> normally distributed?), and moreover is too sensitive to sample size.
> A non-significant test might just mean a small sample, while a 
> minuscule difference in means might be "significant" in a very large 
> sample.
> Standardized differences of means seem to be more accepted, although 
> for continuous covariates comparing actual distributions may be most 
> persuasive.
> From your message it is not clear what type of matching you used and 
> whether all the variables are continuous or some are categorical. If 
> you question the test statistics calculation, you can run the t-test 
> manually and check. While I'm not knowledgeable about practices in 
> sociology and economics, you may want to look at SDM's rather than the 
> pstest results. See e.g.:
> Austin, P. C. (2009). "Balance diagnostics for comparing the 
> distribution of baseline covariates between treatment groups in 
> propensity-score matched samples." Statistics in Medicine 28(25):
> 3083-3107.
> 
> Good luck,
> Adam Olszewski



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/