----- Original Message -----
From: "joachim Wagner" <[email protected]>
To: <[email protected]>
Sent: Monday, August 25, 2003 2:46 AM
Subject: st: How much can we trust Stata's non-linear solver(s)?
> Dear Listmembers,
>
> at the weekend I read a paper by McCullough and Vinod on "Verifying the
> solution from a nonlinear solver: A case study" in the June 2003 issue of
> the American Economic Review. For those of you who are not in economics,
> this is an absolute top-journal in our discipline, and the authors are
> considered to be top-experts in the field. I was shocked (really, believe
> me!) to learn how much can go wrong and goes wrong when one tries to find
> the ML solution of such day-to-day problems as the parameters of a probit
> model using a PC and several programs. The authors give examples, and they
> argue that "the researcher's job is not done when the program reports
> convergence - it is only beginning" (p. 876), and ask us to examine the
> gradient, inspect the solution path, evaluate the Hessian, including an
> eigensystem analysis, and profile the likelihood. Uff. Do I really have to
> learn how to do all this? Do I have to use three or more different packages
> to compare the solutions? Shall I demand this when I sit down to write my
> next referee report later this week? Must I be aware that the next report I
> receive will demand me to do all this?
>
> Unfortunately, the authors do not name horses and riders (as we say in
> Germany - in German, of course), so I have no idea whether Stata is among
> the programs one can trust or not. Obviously, I want to know, and I am sure
> many of the list-members share my view. Given that the acknowledgement
> footnote of the paper mentions W. Gould, I hope he can tell us more.
>
> From a slightly different perspective, would it be a good idea to
> implement the steps suggested by the authors for a "post-convergence" check
> in Stata ?
>
> Given that I am no expert in numerical computing and all the related
> difficulties, I would like to learn your opinions, experiences etc.
>
> Best,
>
> Joachim
>
For what it is worth, when McCullough and Vinod write (referring to a paper by
Houston Stokes "On the Advantage of Using Two or More Econometric Software
Systems to Solve the Same Problem")
"Observe that nonexistence was no impediment to five solvers declaring not only
that a solution existed, but that the solution had been located. For sake of
completeness we note that another two packages did refuse to produce an answer
because they recognized that the maximum likelihood estimator did not exist--the
developers of these packages had written their programs to examine the data for
possible signs of nonexistence of the solution." (McCullough and Vinod, page
873)
One of the two statistical packages that recognized that a solution did not
exist was Stata, the other was TSP.
A preprint copy of Stokes' can be found (in Word format) at
http://tigger.uic.edu/~hhstokes/papers.html
In the paper, while Stokes finds that Stata recognizes that a solution does not
exist, he does find Stata's results 'strange and confusing' -- but I am not sure
I follow.
He writes:
"As a final test of packages, Stata version 7.0 for Linux was tried. Stata
reported "note: d2~=0 predicts success perfectly. d2 dropped and 15 obs not
used." The Stata results are shown in Table 2, column 5. The reported
coefficients and log likelihood ( -8.64038) for this "reduced model" were found
to be nearly the same as was found by the other systems for the complete period
including D2. What Stata appeared to have done is to just not report the
coefficients for D2. The cryptic Stata message night cause some confusion to the
average researcher since what Stata reports is not Maddala's model (2), which is
a probit model estimated for all 44 observations without D2 on the right hand
side. If the Maddala equation (2) is run, all programs replicate the results
reported by Maddala." (page 4.)
"Sorting the dataset with respect to D1, the dependent variable, reveals that
when D1=0, D2 is always 0 but then D1 =1, D2 can be 0 or 1. This means that if
we find that D2 =1, we know for certain that D1 = 1. There were 15 such cases.
Once the sort was studied, the formerly cryptic message in Stata was understood.
All 15 such cases were detected by Stata. What is strange is that if Stata
really believed its own message, it would have either reported nothing (the best
decision) or reported the subset model in equation (2). Just not reporting the
D2 coefficient but leaving all other coefficients in the model is a strange and
confusing choice. " (page 6)
"What started as a replication of probit results reported in Maddala [16] was
extended when different results were obtained with the software systems B34S,
SAS, RATS, LIMDEP and Stata. The differences were traced to the effect of
alternative convergence tolerances on both the rank of the variance covariance
matrix and the point estimate of one variable that was caused by Quasicomplete
Separation. Only two software systems, Stata and TSP, were able to detect the
underlying problem that for this model and these data, the maximum likelihood
estimate does not exist. " (page 12)
Scott
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
