Re: st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow?

[simone ferro <simone.ferro88@gmail.com>]

Thank you so much for your answer,
it was very clear and helpful,
I followed your suggestion and I got better results, so thank you very much,
by the way I'm using a kernel matching based on a probit regression,
variables are mainly continuous.
Now I get a better balance, but as expected t-tests refuse the null-hypotesis of equal means,
and chi2 test refuse the null hypotesis of balance as well,
looking at the means it seems to me that this is a good balance,
do you think I can simply don't look at t-tests and chi2 test?
---------------------------------------------------------
             |       Mean               |     t-test
    Variable | Treated Control    %bias |    t    p>|t|
-------------+--------------------------+----------------
cost_per_e~l |  60.95   60.053      4.4 |  -3.48  0.001
asset        | 193.35   260.33     -5.3 |  -1.41  0.159
employees    | 398.82   551.06     -7.9 |  -0.97  0.333
av_age       | 54.424   54.951     -6.9 |   1.69  0.092
donne_perc   | .29114   .28214      4.3 |   3.86  0.000
laureati_p~c | .23952   .23494      1.9 |   0.11  0.914
bluecollar~c | .35553   .37312     -5.8 |  -2.94  0.004
occupaz_min  | .16463   .13253      9.1 |  -0.13  0.895
donne_perc   | .29114   .28214      4.3 |   3.86  0.000
----------------------------------------------------------
Pseudo R2      LR chi2        p>chi2      MeanB     MedB
----------------------------------------------------------
    0.163        86.83         0.000       5.6       5.3
----------------------------------------------------------

thank you again,
Regards,
Simone

Il giorno 13/gen/2013, alle ore 05:18, Adam Olszewski <adam.olszewski@gmail.com> ha scritto:

> Hi Simone,
> t-test based comparisons after PS matching are highly controversial.
> t-test makes a lot of usually untenable assumptions (are the variables
> normally distributed?), and moreover is too sensitive to sample size.
> A non-significant test might just mean a small sample, while a
> minuscule difference in means might be "significant" in a very large
> sample.
> Standardized differences of means seem to be more accepted, although
> for continuous covariates comparing actual distributions may be most
> persuasive.
> From your message it is not clear what type of matching you used and
> whether all the variables are continuous or some are categorical. If
> you question the test statistics calculation, you can run the t-test
> manually and check. While I'm not knowledgeable about practices in
> sociology and economics, you may want to look at SDM's rather than the
> pstest results. See e.g.:
> Austin, P. C. (2009). "Balance diagnostics for comparing the
> distribution of baseline covariates between treatment groups in
> propensity-score matched samples." Statistics in Medicine 28(25):
> 3083-3107.
> 
> Good luck,
> Adam Olszewski
> 
> On Sat, Jan 12, 2013 at 7:37 AM, simone ferro <simone.ferro88@gmail.com> wrote:
>> dear Statalist,
>> 
>> I would please need some clarifications about the interpretation of the command pstest after running psmatch2:
>> I report a random output just as an example.
>> 
>> pstest ebitda_marg asset employees av_age donne_perc laureati_perc bluecollar_perc, both
>> 
>> ------------------------------------------------------------------------------
>>                Unmatched |       Mean               %reduct |     t-test
>>    Variable      Matched | Treated Control    %bias  |bias| |    t    p>|t|
>> --------------------------+----------------------------------+----------------
>>  ebitda_marg   Unmatched | 11.404   8.2304     36.8         |   3.25  0.001
>>                 Matched  | 10.746   10.555      2.2    94.0 |  -2.33  0.020
>>                          |                                  |
>>        asset   Unmatched | 395.52   34.389     28.6         |   2.48  0.014
>>                 Matched  | 115.36   89.157      2.1    92.7 |  -2.10  0.037
>>                          |                                  |
>>    employees   Unmatched | 641.98   508.48      4.6         |   0.42  0.677
>>                 Matched  | 474.12   704.43     -8.0   -72.5 |   0.18  0.857
>>                          |                                  |
>>       av_age   Unmatched | 53.369   56.714    -45.0         |  -4.06  0.000
>>                 Matched  |  53.39   53.051      4.6    89.9 |   3.84  0.000
>>                          |                                  |
>>   donne_perc   Unmatched | .34711   .37372    -11.8         |  -1.06  0.291
>>                 Matched  | .34805   .33374      6.3    46.2 |   0.16  0.874
>>                          |                                  |
>> laureati_perc   Unmatched | .26656   .18165     35.4         |   3.15  0.002
>>                 Matched  | .26815   .23665     13.2    62.9 |  -1.52  0.129
>>                          |                                  |
>> bluecollar_~c   Unmatched | .30019    .3349    -11.2         |  -1.00  0.317
>>                 Matched  | .30425   .33592    -10.2     8.7 |  -0.28  0.776
>>                          |                                  |
>> ------------------------------------------------------------------------------
>> If I understood well, reported t-.tests' null hypothesis is that the two covariates are equal in treated and control group,
>> so I should look at t-tests to check if the groups are well balanced,
>> In some tutorial instead I've read that the right approach is to look at the bias percentage that should be under 10 to be considered ok,
>> 
>> Which one of the two approaches is the right one?it's fundamental for me to understand because they provide totally different interpretations.
>> indeed if I look at t-tests, I find problems with ebitda_margin, asset and av_age, because they are significantly different in the two groups,
>> while if I look at the bias percentage, I find problem with laureati_perc and blecollar, because their bias% are bigger than 10.
>> 
>> I also would appreciate your confirmation of the interpretation of the two indicators(%bias and t-tests), because with this interpretation I find the two values contradictory.
>> looking for example at the variable av_age, I find a very little bias, and the means after matching of treated and control group are almost identical(53.39 and 53.051), by the way the t-test reports a p-value of 0.000!
>> So it seems like I have misunderstood the meaning of the t.test because I don't think that 53.39 and 53.051 can be statistically different with a t-stat of 3.84, also given the nature of the variable(the average age of the managers of a firm) that should infact be quite variable.
>> The same happens with the variable ebitda, which reports a bias% of 2.2% and almost identical values(10.746 and 10.555), but t-stat is -2.33 and p-value 2%!
>> Can you please help me?
>> thanks in advance for the help,
>> Regards,
>> Simone Ferro
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/