Re:st: RE: psmatch2 and p-value

["Ariel Linden, DrPH" <ariel.linden@gmail.com>]

Hi Collette,

First, to the tables: 

The first line indicates what the unmatched/unadjusted values look like
(ie., comparison of all the treated to all of the untreated). Thus the
t-stat indicates this is not significant (t-stat of 1.96 equals p-value of
0.05, so you are looking for t-stat values equal to or greater than 1.96). 

The second line shows the average treatment effect on the treated: here, the
140 matched treated are compared to the 140 matched untreated. Here the
t-stat is even lower (-0.97), suggesting there is no statstically
significant difference in RAW between matched treated and controls.

The table below that indicates that there were 140 matched pairs, and 293
treated subjects for which controls could not be found (because they were
beyond the common support range).

Conceptual issues:


What I don't see here is what covariates you used to generate the pscore, or
if balance was achieved on those covariates? This is perhaps the most
important aspect of observational studies, and should not be disregarded.

The next issue  (and my biggest area of concern) is that you are running
this algorthm for each year and then adding up the scores in the end. I am
not sure I agree with that. These are the same firms, and so there is going
to be repeated measurements on the same firms over time. Your methodology
does not account for the longitudinal nature of the data and the
auto-correlation that would be expected within firm.

A more appropriate methodology would be to generate propensity score based
weights for the baseline period (preintervention) for each group in the
dataset, and then use those weights within a longitudinal model (ie., GEE
with the appropriate family and link).

These are not trivial issues, and I suggest you consult a statistician or
read the appropriate literature on these topics (longitudinal data, causal
inference from observational data, etc.).

I hope this helps

Ariel


Date: Tue, 30 Nov 2010 16:12:35 -0000
From: "Colette.Grey" <Colette.Grey@ul.ie>
Subject: st: RE: psmatch2 and p-value 

Hi Ariel,

I appreciate you taking time to help me with these queries, I am learning
and making some progress I hope. I have two questions about the output from
psmatch2.

I have data for three years so I want to estimate the propensity score
annually and match the firms annually and am using psmatch2 for this. Then I
use the results from the three years together to asses the ATT. Yes, I do
have more treatment than control firms. I am starting by just looking at the
matched firms.

(1) What I have is a result below where RAW is the outcome of interest and
TARG is the treatment:


. psmatch2 TARG, outcome (RAW) pscore (ps2) logit neighbor(1) noreplacement
-
----------------------------------------------------------------------------
------------
        Variable     Sample |    Treated     Controls   Difference        
S.E.   T-stat
-
----------------------------+-----------------------------------------------
------------
         RAWAWCA  Unmatched | .005222726   .015996705  -.010773979  
.006727021    -1.60
                        ATT | .007248377   .015996705  -.008748328 
 .009040694    -0.97
-
----------------------------+-----------------------------------------------
------------
Note: S.E. does not take into account that the propensity score is
estimated.

What I don't know form the above is whether or not the T-stat is
significant, I need a p-value. Do you know please how I can get a p-value?


(2) The below is also printed when I run the above command:


psmatch2: |   psmatch2: Common
 Treatment |        support
assignment | Off suppo  On suppor |     Total
- -----------+----------------------+----------
 Untreated |         0        140 |       140 
   Treated |       293        140 |       433 
- -----------+----------------------+----------
     Total |       293        280 |       573

My question is if the above is calculated on the 280 matched firms or the
573 firms?
How do I tell Stata which I want, assuming I am able to do so please?

Any help would be great, thanks,

Colette 


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/