RE: st: Handbook on impact evaluation with Stata examples

[Adam Collins <ACollins@fxb.org>]

Dear all, 

Since it is evident that there are some fairly negative viewpoints on this specific text, I wondered if I may change the direction of this discussion somewhat and ask you all what texts you DO recommend for impact evaluations using Stata.

Judging from the bestsellers in the Stata bookstore, the following texts appear to be contenders:

 - Statistics with Stata (Updated for Version 10)  - Lawrence C. Hamilton
 - Applied Survey Data Analysis  - Steven G. Heeringa, Brady T. West, and Patricia A. Berglund
 - A Short Introduction to Stata for Biostatistics  - Michael Hills and Bianca L. De Stavola

But what does the Statalist community think of these texts (and others?)

Thanks!

Adam


----------------------------
Adam Collins 
Program Performance and Evaluation Officer 

FXB International 
44, rue de Lausanne 
CH - 1201 Genève 
SWITZERLAND
Tel      +41 22 741 00 31
Fax     +41 22 731 17 57
www.fxb.org
----------------------------



-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Christopher Baum
Sent: 02 October 2010 22:11
To: statalist@hsphsun2.harvard.edu
Subject: re: st: Handbook on impact evaluation with Stata examples

<>
To add to Eric de Souza's comments, Chapter 6 on instrumental variables methods introduces these methods as involving running separate first-stage and second-stage regressions, which as we know will not yield consistent standard errors or RMS error. They call an exactly-identified equation the 'two-stage least squares' approach. Normally when teaching this material we make the distinction that IV in the narrow sense applies when you have one endogenous regressor and one instrument, and 2SLS is needed when there are more instruments than endogenous regressors. They fail to clarify this distinction.

As Eric mentioned, the authors horribly misuse the term 'weak instruments' to mean 'invalid instruments', those which cause an overid test to fail. They fail to name the overid test as a Sargan or Hansen test, and are oblivious to the issue that the Sargan test will apply with iid errors, but a Hansen test should be used otherwise. As they have not to this point mentioned non-iid errors in the chapter, that is perhaps not surprising.

They cite a good deal of literature, but fail to mention Angrist and Pischke's Mostly Harmless Econometrics book. Reading the IV chapters of that book would be far more useful for the practitioner than wasting their time with this publication.

In the Stata chapter on IV, chapter 15, we are treated to the notion that the authors are running Stata version 6 or 7 at best. They start off by:

1) creating a number of interaction variables, not with factor variables, but with the -for- command! How many people remember the -for- command?  All right, not everyone in the third world has Stata 11, but if they have Stata, I bet they have -foreach-.

A clue to their use of antiquated commands is the screenshot of the Stata environment on p.148, which illustrates a Stata 8.0 screen. Not Stata 8.2, mind you. Why should a document with a 2010 copyright refer to software as of 2003?

2) estimating an instrumental variables model with the -ivreg- command. which renders difficult the evaluation of overidentification, endogeneity, etc. -ivregress- has been around for a while. -ivreg2- of Baum,Schaffer,Stillman has been around longer than that. Both provide a wealth of capabilities lacking from -ivreg-. Furthermore, they do not even understand the syntax of  -ivreg-, as they think that included exogenous variables must be repeated in the parentheses, e.g.,

ivreg lexptot agehead-educhead lnland vaccess pcirr rice-oil (dfmfd= agehead-educhead lnland vaccess pcirr rice-oil fch*), first;

This does no harm, but if you're teaching people how to use the command, why give them this message?  They do the same thing with -xtivreg-. 

3) Strangely, given that they use -ivreg- rather than -ivreg2-, they then suggest using -ivendog-, part of the B-S-S Stata Journal package containing -ivreg2-! They fail to mention that -ivendog- is not part of oficial Stata. "Stata has a command 'ivendog' that performs an F-test..."  Sorry, it doesn't, and never has had an official command of that name.

Given this comedy of errors on a fairly straightforward topic such as IV, I cringe at the thought of what they may have done to topics like propensity score matching, regression discontinuity, etc.  On p.168 they indicate that "the propensity score matching technique is implemented by an .ado file called pscore.ado." They fail to indicate that this is a user-written command (published a long time ago in the Stata Journal), nor do they recognize that most users these days would look for -psmatch2- of Leuven and Sianesi. The -pscore- command has not been updated since 2005.