st: RE: re: St : matching estimator

["Millimet, Daniel" <millimet@mail.smu.edu>]

1. My rough opinion with respect to the first question is that the metric - say Mahalanobis distance versus propensity score difference - is much less significant than the choice over (a) what variables are used to construct the distance measure (including the decision over higher order terms and interaction terms involved the matched on variables) and (b) what estimator is actually used once distance measures are chosen (e.g., nearest neighbor, kernel matching, etc.)

2. No one can judge whether is unobserved heterogeneity without knowing the details of the application and the covariates included in the model.  If you are worried, there are methods designed to assess sensitivity to the unconfoundedness assumption, such as Rosenbaum bounds or the method developed in the user-written -sensatt- command or the methods discussed in Altonji et al. (J Political Economy, 2005).

Best,
Dann

****************************************************
Daniel L. Millimet
Research Fellow, IZA
Professor, Department of Economics
Box 0496
SMU
Dallas, TX 75275-0496
phone: 214.768.3269
fax: 214.768.1821
web: http://faculty.smu.edu/millimet
****************************************************


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Ayman Farahat
Sent: Tuesday, January 03, 2012 11:32 AM
To: statalist@hsphsun2.harvard.edu
Subject: st: re: St : matching estimator






Thanks for the list. To give more context i am trying to evaluate a treatment effect (users self selecting to download an app. If they don't download the app, they can still access the service through the web interface). I am trying to estimate the causal impact of the treatment (downloading the app) on the user engagement (number of times they access the service, churn rate ... etc). 

I had two concerns about the matching estimator :
1) The sensitivity to the metric used to find matching subjects. 

2) The risk of unobserved heterogeneity specially given the self selection. 

Thanks
Ayman



________________________________
From: Cameron McIntosh <cnm100@hotmail.com>
To: STATA LIST <statalist@hsphsun2.harvard.edu> 
Sent: Thursday, December 29, 2011 12:52 PM
Subject: RE: st: re: St : matching estimator

I will add a list of references that you can cull from, many are directly relevant to your case, I am sure (others may be interested as well):

Glynn, A.N., & Quinn, K.M. (2010).  An Introduction to the Augmented Inverse Propensity Weighted Estimator. Political Analysis. 18(1):36-56.

Glynn, A.N., & Quinn, K.M. (January 23, 2008). Choosing an Identifying Set of Matching or
Conditioning Variables. http://scholar.harvard.edu/sites/scholar.iq.harvard.edu/files/aglynn/files/Choosing_an_Identifying_Set.pdf

Steiner, P.M., Cook, T.D., & Shadish, W.R. (2011). On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores. Journal of Educational and Behavioral Statistics, 36(2), 213-236.
Trojano, M., Pellegrini, F., Paolicelli, D., Fuiani, A., & Di Renzo, V. (2009). Observational Studies: Propensity Score Analysis of Non-randomized Data. The International MS Journal, 16, 90–97.http://www.msforum.net/journal/download/20091690.pdf

Li, L. (May 13, 2011). Propensity Score Analysis with Matching Weights. COBRA Preprint Series, Article 79. http://biostats.bepress.com/cobra/ps/art79
Fan, X., & Nowell, D.A. (2011). Using Propensity Score Matching in Educational Research. Gifted Child Quarterly, 55(1), 74-79.

Austin, P.C. (2008a). A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Statistics in Medicine, 27(12), 2037–2049.http://www.epi.msu.edu/janthony/requests/propensity/Austin_A%20critical%20appraisal%20of%20propensity%20score.pdf
*** the above paper is followed by numerous commentaries 
Williamson, E., Morley, R., Lucas, A., & Carpenter, J. (2011). Propensity scores: From naive enthusiasm to intuitive understanding. Statistical Methods in Medical Research, Online First.

Pruzek, R.M. (2011).
Introduction to the Special Issue on Propensity Score Methods in Behavioral Research. Multivariate Behavioral Research, 46(3), 389-398.

Austin, P.C. (2011). An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behavioral Research, 46(3), 399-424.http://www.tandfonline.com/doi/pdf/10.1080/00273171.2011.568786

Kelcey, B. (2011). Covariate Selection in Propensity Scores Using Outcome Proxies. Multivariate Behavioral Research, 46(3), 453-476.

Hill, J., Weiss, C., & Zhai, F. (2011). Challenges With Propensity Score Strategies in a High-Dimensional Setting and a Potential Alternative. Multivariate Behavioral Research, 46(3), 477-513.

Thoemmes, F.J., & West, S.G. (2011). The Use of Propensity Scores for Nonrandomized Designs With Clustered Data. Multivariate
Behavioral Research, 46(3), 514-543.

Lottridge, S.M., Nicewander, W.A., & Mitzel, H.C. (2011). A Comparison of Paper and Online Tests Using a Within-Subjects Design and Propensity Score Matching Study. Multivariate Behavioral Research, 46(3), 544-566.

Coffman, D.L. (2011). Estimating Causal Effects in Mediation Analysis Using Propensity Scores. Structural Equation Modeling, 18(3), 357-369.

Jo, B., Stuart, E.A., MacKinnon, D.P., & Vinokur, A.D. (2011). The Use of Propensity Scores in Mediation Analysis. Multivariate Behavioral Research, 46(3), 425-452.
Stuart, E.A. (2010). Matching Methods for Causal Inference: A Review and a Look Forward. Statistical Science, 25(1), 1-21.
http://arxiv.org/PS_cache/arxiv/pdf/1010/1010.5586v1.pdf

Spector, P.E., & Brannick, M.T. (2011). Methodological Urban Legends: The Misuse of Statistical
Control Variables. Organizational Research Methods, 14(2), 287-305.

Stürmer, T., Rothman, K.J., Avorn, J., & Glynn, R.J. (2010). Treatment Effects in the Presence of Unmeasured Confounding: Dealing With Observations in the Tails of the Propensity Score Distribution—A Simulation Study. American Journal of Epidemiology, 172(7), 843-854.

Westreich, D., Cole, S.R., Funk, M.J., Brookhart, M.A., & Stürmer, T. (2011). The role of the c-statistic in variable selection for propensity score models. Pharmacoepidemiology and Drug Safety, 20(3), 317–320.
Waernbaum, I. (2010). Propensity score model specification for estimation of average treatment effects. Journal of Statistical Planning and Inference, 140(7), 1948-1956. 

Thoemmes, F.J., & Kim, E.S. (2011). A Systematic Review of Propensity Score Methods in the Social Sciences. Multivariate Behavioral Research, 46(1), 90-118,

Austin, P.C. (2011). A Tutorial and
Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality. Multivariate Behavioral Research, 46(1), 119-151.
Heinrich, C., Maffioli, A., & Vázquez, G. (2010). A primer for applying propensity-score matching. Impact-Evaluation Guidelines, Technical Notes No. IDB-TN-161. Office of Strategic Planning and Development Effectiveness, Inter-American Development Bank.http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=35320229

Harder, V.S., Stuart, E.A., & Anthony, J.C. (2010). Propensity Score Techniques and the Assessment of Measured Covariate Balance to Test Causal Associations in Psychological Research. Psychological Methods, 15(3), 234-249. 
Steiner, P.M., Cook, T.D., Shadish, W.R., & Clark, M.H. (2010). The Importance of Covariate Selection in
Controlling for Selection Bias in Observational Studies. Psychological Methods, 15(3), 250-267. 
Sekhon, J.S. (2011). Multivariate and propensity score matching software with automated balance optimization: the Matching package for R. Journal of Statistical Software, 42(7).http://sekhon.berkeley.edu/papers/MatchingJSS.pdf

Sekhon, J.S. (May 20, 2011). Multivariate and Propensity Score Matching with Balance Optimization. Package ‘Matching’, Version 4.7-14.http://cran.r-project.org/web/packages/Matching/Matching.pdfhttp://cran.r-project.org/web/packages/Matching/index.html

Cheers,
Cam
> From: ariel.linden@gmail.com
> To: statalist@hsphsun2.harvard.edu
> Subject: st: re: St : matching estimator
> Date: Thu, 29 Dec 2011 11:58:02 -0500
> 
> Hi Ayman,
> 
> It is not clear from your post exactly what you are asking for, or trying to
> do with your data? 
> 
> The propensity score is typically used as the matching variable when you
> have a very large number of variables. You would find the appropriate
> approach that minimizes the distance of the propensity score within matches.
> Note that just because you have close matches on the propensity score, does
> not ensure you have balance on the underlying covariates. Thus, it is more
> important that you check for covariate balance than have balance on the
>
propensity score.
> 
> If you prefer to match on the covariates themselves, you have several
> options. The most ubiquitous approach is mahalanobis distance matching. This
> can be done within an existing user written program such as -psmatch2- or
> -optmatch2-, or - mahapick-.
> 
> You could also consider -cem- which is another user written program that
> uses stratification to bucket units.
> 
> My personal preference is to use a propensity score weighting approach
> (IPTW, ATT, or ATE weights), and then model the outcome using that weight.
> You could also consider kernel weighting, which is one of the options within
> -psmatch2-.
> 
> So basically you have a lot of options, but if you'd like a more specific
> response, you'll have to ask your question in a more specific manner... 
> 
> I hope this helps
> 
> Ariel
> 
> 
> Date: Wed, 28 Dec 2011 12:22:58 -0800
> From: Ayman Farahat <ayman.farahat@yahoo.com>
> Subject: st: St : matching estimator
> 
> Hello
> I am looking into using nmatch to evaluate treatment effect.
> However, i am not sure if there is a natural meteric to find closest match.
> Could the choice of meteric impact résult?
> Any references to work comparing matching estimator to propensity or Heckman
> greatly appreciated.
> Thanks
> Ayman
> Sent from my iPad
> 
> 
> *
> *   For searches and help try:
> *  http://www.stata.com/help.cgi?search
> *  http://www.stata.com/support/statalist/faq
> *  http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*  http://www.stata.com/help.cgi?search
*  http://www.stata.com/support/statalist/faq
*  http://www.ats.ucla.edu/stat/stata/   ;                       

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/