Re: Re: Re: st: replicating 2 X 2 data from a paper

[David Hoaglin <dchoaglin@gmail.com>]

Clyde,

Thanks for your perspective.

I was criticizing only rough edges and errors that I found in the
articles themselves.  The absence of adequate documentation is another
matter entirely.  For some time, the guidelines of the International
Committee of Medical Journal Editors have advised that the statistical
methods should be documented in sufficient detail that a knowledgeable
person with access to the original data could reproduce the results.
I agree that published articles hardly ever have space to do this, and
the gap has widened as more-sophisticated methods have come into use.
But, as you point out, an online appendix provides all the space
needed.  (Some months ago I read an article that had appeared in 2010
in the New England Journal of Medicine.  One appendix was 150 pages
long, and the protocol for the study was in a separate appendix.)  It
seems to me that authors no longer have any excuse for not documenting
their work adequately.

Taking Nick's comment about a "smokescreen" a step further, one
article that I read introduced its own statistical technique, with no
pedigree in the statistical literature.  In support of that technique,
the article cited four papers, not one of which provided any support
for the authors' approach.  Indeed, one of the papers said that
approaches like the authors' should not be used.  My conclusion was
that the authors had not read any of those papers and were citing them
to deceive reviewers.  I was not a reviewer of that article; it had
been published in a "peer-reviewed" journal.

David Hoaglin

On Thu, Apr 4, 2013 at 12:49 PM, Clyde B Schechter
<clyde.schechter@einstein.yu.edu> wrote:
> In response to a question from Ariel Linda, David Hoaglin wrote, mostly as an afterthought:
>
> "As a statistician, I think you expect too much of the "statistical
> reviewers" or of peer review more generally.  Many papers that I read
> contain rough edges and errors that, to my amazement, the authors and
> the reviewers somehow failed to notice."
>
> Here's my two cents on this generic issue.  Nearly all the journals I have dealth with have stringent word limits for articles.  It is almost never the case that a truly satisfactory description of the methods and important results can be accommodated within those limits, and when the time comes to cut, it is usually the methods that get the meat ax, at least in clinincal journals.
>
> It may well be that the authors provided a fuller explanation to the reviewers "behind the scenes" but it never made it into the article because of space limitations.  I have experienced this phenomenon myself both as an author and as a reviewer.
>
> My hope and expectation is that as publication moves away from paper and comes to rely more on electronic formats these word limits, relics attributable to the expense of paper, will disappear (or at least greatly relax) and that on-line articles will include much better descriptions of methods.  It seems to be happening already: I have encountered several articles where the hard copy gives the "highlights" of the methods, but an "on-line appendix" has all the detail one could ask for.
>
>
> Clyde Schechter
> Dept. of Family & Social Medicine
> Albert Einstein College of Medicine
> Bronx, NY, USA

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/