st: RE: Loosen precision for ttest

[Joe Canner <jcanner1@jhmi.edu>]

Ryan,

Perhaps someone else can explain why Stata is doing this and whether it can be fixed (I suspect not), but in the meantime I would suggest one of the following:

1. Re-consider whether you should be using propensity scores if you have many strata with the same score.  Perhaps some other matching method (e.g., coursened exact matching using -cem-) would work better and perhaps be more appropriate.  Without knowing the number and nature of variables you are matching on it is difficult to give more specific advice.

2. Consider using -psmatch2- instead of -pscore-.  -psmatch2- uses similar methods to generate the score, but different methods to check for balance.  It also does the matching as part of the main run rather than delegating it to another program as -pscore- does.

3. If -pscore- is your only option, you might be able to trick it into behaving by using the -numblo()- option to specify a different starting number of blocks.

Regards,
Joe Canner
Johns Hopkins University School of Medicine
________________________________________
From: owner-statalist@hsphsun2.harvard.edu [owner-statalist@hsphsun2.harvard.edu] on behalf of Ryan Turner [rjturner@cmu.edu]
Sent: Tuesday, September 03, 2013 7:21 PM
To: statalist@hsphsun2.harvard.edu
Subject: st: Loosen precision for ttest

Dear Statalist:

I have a precision question; I know much has been written about it (http://blog.stata.com/tag/precision/) but as yet I am not able to resolve this issue.

I am using -pscore- to perform propensity score matching, and in the first step it checks that the estimated propensity score is equal across treatment and non-treatment groups using -ttest-.  My propensity score specification is based on dummies, resulting in several swaths of the same estimated pscore.  -pscore- should calculate the same number of blocks as the number of unique pscores ('swaths').  But instead, because of this ttest below, it splits the block and retests the same observations, ad infinitum.

In words, the algorithm determines pscore to be non-constant in the last block, where 'non-constant' means 'differs by floating point error'.  Therefore, I believe I need to 'loosen' the precision in this ttest.

Any thoughts are appreciated.  This issue only appears under certain specifications (which, unfortunately, are the ones I believe I want), and not in all blocks with a constant pscore.

Best,
Ryan


******************************************************
Step 1: Identification of the optimal number of blocks
Use option detail if you want more detailed output
******************************************************

(79 blocks, mostly empty, omitted)

Test in block 80

Observations in block 80
 obs: 16453,  control: 407,  treated: 1676


Test for block 80

Two-sample t test with equal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |     407    .8046087    5.51e-18    1.11e-16    .8046087    .8046087
       1 |    1676    .8046087           0           0    .8046087    .8046087
---------+--------------------------------------------------------------------
combined |    2083    .8046087    1.52e-18    6.94e-17    .8046087    .8046087
---------+--------------------------------------------------------------------
    diff |           -1.11e-16    2.71e-18               -1.16e-16   -1.06e-16
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t = -40.9193
Ho: diff = 0                                     degrees of freedom =     2081

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.0000         Pr(|T| > |t|) = 0.0000          Pr(T > t) = 1.0000



The mean propensity score is different for
treated and controls in block 80
Split the block 80 and retest





For completeness I include a successful ttest from a different specification, where the mean is a rather boring 0 and all p-values rejecting the null are missing (.).  So this obviously doesn't have the floating point precision issue.

Two-sample t test with equal variances
------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       0 |     444           0           0           0           0           0
       1 |    5829           0           0           0           0           0
---------+--------------------------------------------------------------------
combined |    6273           0           0           0           0           0
---------+--------------------------------------------------------------------
    diff |                   0           0                       0           0
------------------------------------------------------------------------------
    diff = mean(0) - mean(1)                                      t =        .
Ho: diff = 0                                     degrees of freedom =     6271

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) =      .         Pr(|T| > |t|) =      .          Pr(T > t) =      .




--
Ryan J. Turner
Engineering and Public Policy
Carnegie Mellon University
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/