RE: st: RE: psmatch2 and p-value

["Colette.Grey" <Colette.Grey@ul.ie>]

Thanks a million Ariel for such a comprehensive respone, I really appreciate the help and guidance.

I ran the pscore command in Stata and it said 'The balancing property is satisfied'. I did notice a question on this list about how many covariates one should use. I get the above even after dropping some of the covariates. In contrast, I have read the more covariates the better. 

When calculating the _pscore, I used psmatch2 in Stata and generated a variable ps2 to be equal to _pscore so from then on, I used the following:

	psmatch2 TARG, outcome (RAW) pscore (ps2) logit neighbor(1) noreplacement

I totally understand your concerns regarding the methodology, that is my next task, I was trying first to come to terms with the commands and outputs in Stata.

Again thanks,

Colette



-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Ariel Linden, DrPH
Sent: 01 December 2010 18:11
To: statalist@hsphsun2.harvard.edu
Subject: Re:st: RE: psmatch2 and p-value 

Hi Collette,

First, to the tables: 

The first line indicates what the unmatched/unadjusted values look like (ie., comparison of all the treated to all of the untreated). Thus the t-stat indicates this is not significant (t-stat of 1.96 equals p-value of 0.05, so you are looking for t-stat values equal to or greater than 1.96). 

The second line shows the average treatment effect on the treated: here, the 140 matched treated are compared to the 140 matched untreated. Here the t-stat is even lower (-0.97), suggesting there is no statstically significant difference in RAW between matched treated and controls.

The table below that indicates that there were 140 matched pairs, and 293 treated subjects for which controls could not be found (because they were beyond the common support range).

Conceptual issues:


What I don't see here is what covariates you used to generate the pscore, or if balance was achieved on those covariates? This is perhaps the most important aspect of observational studies, and should not be disregarded.

The next issue  (and my biggest area of concern) is that you are running this algorthm for each year and then adding up the scores in the end. I am not sure I agree with that. These are the same firms, and so there is going to be repeated measurements on the same firms over time. Your methodology does not account for the longitudinal nature of the data and the auto-correlation that would be expected within firm.

A more appropriate methodology would be to generate propensity score based weights for the baseline period (preintervention) for each group in the dataset, and then use those weights within a longitudinal model (ie., GEE with the appropriate family and link).

These are not trivial issues, and I suggest you consult a statistician or read the appropriate literature on these topics (longitudinal data, causal inference from observational data, etc.).

I hope this helps

Ariel


Date: Tue, 30 Nov 2010 16:12:35 -0000
From: "Colette.Grey" <Colette.Grey@ul.ie>
Subject: st: RE: psmatch2 and p-value 

Hi Ariel,

I appreciate you taking time to help me with these queries, I am learning and making some progress I hope. I have two questions about the output from psmatch2.

I have data for three years so I want to estimate the propensity score annually and match the firms annually and am using psmatch2 for this. Then I use the results from the three years together to asses the ATT. Yes, I do have more treatment than control firms. I am starting by just looking at the matched firms.

(1) What I have is a result below where RAW is the outcome of interest and TARG is the treatment:


. psmatch2 TARG, outcome (RAW) pscore (ps2) logit neighbor(1) noreplacement
-
----------------------------------------------------------------------------
------------
        Variable     Sample |    Treated     Controls   Difference S.E.   T-stat
-
----------------------------+-------------------------------------------
----------------------------+----
------------
         RAWAWCA  Unmatched | .005222726   .015996705  -.010773979
.006727021    -1.60
                        ATT | .007248377   .015996705  -.008748328
 .009040694    -0.97
-
----------------------------+-------------------------------------------
----------------------------+----
------------
Note: S.E. does not take into account that the propensity score is estimated.

What I don't know form the above is whether or not the T-stat is significant, I need a p-value. Do you know please how I can get a p-value?


(2) The below is also printed when I run the above command:


psmatch2: |   psmatch2: Common
 Treatment |        support
assignment | Off suppo  On suppor |     Total
- -----------+----------------------+----------
 Untreated |         0        140 |       140
   Treated |       293        140 |       433
- -----------+----------------------+----------
     Total |       293        280 |       573

My question is if the above is calculated on the 280 matched firms or the
573 firms?
How do I tell Stata which I want, assuming I am able to do so please?

Any help would be great, thanks,

Colette 


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/