re: Re: st: Significance test with -pscore- and -attnd-

["Ariel Linden, DrPH" <ariel.linden@gmail.com>]

Lucas,

You should be aware that if you are trying to " compare these results with
"simple" ATT results" you will necessarily get different results. The
weighting approach provides an ATE estimate, and the attnd will provide you
with a ATT estimate that will likely differ from other ATT estimates because
it strictly uses matching with replacement, and of course, depends on the
k:1 scenario you set up.

All that said, I believe that you'll have to set up your data as wide in
order to use -attnd- (or for that matter, all other "canned" matching
programs available in Stata). Perhaps one work around would be to run such a
program limiting it to time==1 in a long format, but that is my speculation
only.

Ariel 

Date: Mon, 12 Aug 2013 10:21:52 +0200
From: Lukas Borkowski <570722@soas.ac.uk>
Subject: Re: st: Significance test with -pscore- and -attnd-

Ariel,

thank you for the advice on how to check the statistical significance of ATT
following the user-written -attnd- command. The reference I was referring to
is Khandker, S. R., Koolwal, G. B. and Samad, H. A. (2010). Handbook on
Impact Evaluation. Quantitative Methods and Practices. The World Bank.
Washington, D. C. Apologies for the inconvenience.

Coming back to estimating ATT using panel data, I am aware that a Difference
in Difference estimation with propensity weights (ps/(1-ps)) could be a
suitable approach. However, I would like to compare these results with
"simple" ATT results, possibly derived with -attnd-. My question is on how
to implement -attnd- in a panel structure. In a two-waved perfectly balanced
panel, is it reasonable to convert my panel into a wide dataset using
-reshape- to run -pscore- using suitable covariates from the first wave and
then -attnd- on the outcome variable in the second wave? 

Thanks for your advice. I really appreciate it!

Best, Lukas

#
Lukas Borkowski
University of London, School of Oriental and African Studies (SOAS)

M: 570722@soas.ac.uk



On 10.08.2013, at 16:39, "Ariel Linden, DrPH" <ariel.linden@gmail.com>
wrote:

> Lucas,
> 
> You are asking some very basic questions that would suggest you need to
> learn more about propensity score matching (or matching in a more broad
> sense). I suggest you read Stuart (2010) and Caliendo & Kopeinig (2008)
for
> general guidance in this area.:
> 
> Here are some more specific responses to your questions, but as you can
see,
> they require you to know what you're asking (and thus, I suggest you read
> the papers)
> 
> - keep only matched observations? 
> 
> * it depends on what you plan on doing next, and how you limit your
> observations for analysis
> 
> - keep only observations within the common support?
> 
> * it depends on what you plan on doing next, and how you limit your
> observations for analysis
> 
> - estimate the ATT with -attnd- using both panel rounds or only the post
> round?
> 
> * This question suggests that you need to learn about the estimators and
how
> you use them. Briefly, the baseline characteristics are used as a
> pre-processing step for matching. Once matches are derived (and you have
> checked for covariate balance), you move on to the analysis stage. In an
> analysis, you evaluate the effect of the "treatment" on the outcome,
> controlling for baseline characteristics. In this case, you have
controlled
> for baseline characteristics via the matching process, so the analysis is
on
> the outcome (in your terminology, the post round).
> 
> - Do I have to convert my dataset from long to wide to estimate the ATT?
> 
> * If you want to use this procedure, then you should have the data in wide
> format. If you have multiple waves, you have other matching and outcomes
> analysis approaches to consider.
> 
> - How can I see whether the ATT is significant (as, for example, shown but
> not explained in Khandker et al., 2010)?
> 
> *First off, you didn't provide us with the complete reference. Should we
> guess, or are you assuming that we all know exactly what you are
> referencing? What if Khandker wrote 10 papers in 2010? 
> 
> *Second, you have (at least) two ways of finding the level of
significance:
> first, you know that p = 0.05 is approximately equal to t = 1.96, so in
this
> case with a t score of > 1.96, you can be reasonably assured that the
value
> is statistically significant. A more specific approach to use, following
> -attnd- (a user written program -findit attnd-) looks something like this
> (after running some example code): 
> 
> . return list
> 
> scalars:
>               r(ncnd) =  12144
>               r(ntnd) =  185
>            r(tsattnd) =  -12.52377338537646
>            r(seattnd) =  587.1491980157692
>              r(attnd) =  -7353.32349935502
> 
> * estimate t(df,t)
> . di t(r(ncnd)+r(ntnd)-2,r(tsattnd))
> 4.567e-36
> 
> I hope this helps
> 
> Ariel
> 
> Stuart, E.A. (2010) Matching methods for causal inference: a review and a
> look forward. Statistical Science, 25(1), 1-21.
> 
> Caliendo, M. Kopeinig, S. (2008) Some practical guidance for the
> implementation of propensity score matching. Journal of Economic Surveys,
> 22, 31-72. 


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/