st: Re: Yet another question regarding Stata Tip #87

[Maarten Buis <maartenlbuis@gmail.com>]

On Mon, Sep 24, 2012 at 11:07 AM, Maarten Buis wrote:
> So
> statistical significance can have a place in a scientific argument,
> but it is not strictly necessary, and it is certainly not the main
> criterium on which to judge the "scientificness" of an argument.

Nick kindly corrected my Greek by pointing out that criterium should
have been criterion.

> (**) It assumes that the only source of uncertainty comes from the
> fact that you have drawn a random sample, so it is possible that you
> have drawn a weird sample by chance. This is (almost) always not the
> only reason why we are uncertain about our results.

To elaborate on that a bit more: Collecting data is often a lot more
messy than the resulting dataset suggests. I mainly work with datasets
obtained from interviewing people, so that is what my examples refer
to. In that case, you need to worry about what the exact question was
that was asked to the respondents, did the respondents understand that
question, did the interviewer understand the answer, what did the
respondent and/or the interviewer do when the answer did not (quite)
fit the answer categories, etc. For example, I worked at a department
where they taped interviews in order to see how messy it can get. They
had a question that translates to something like: "How many days of
the week do you consume grain products for breakfast?" The idea of
course was that this would capture things like bread, cereal, müsli,
etc. One respondent answered "None ... hmmm ... Well, beer is made of
grain too, isn't it?"

These are issues that can, and will, occur even when both the
respondent and interviewer are genuinely trying to be helpful. Things
get even more messy when the respondents get bored half way trough the
interview or when the interviewers need to push trough as many
interviews as possible in order to get a reasonable wage...

-- Maarten

---------------------------------
Maarten L. Buis
WZB
Reichpietschufer 50
10785 Berlin
Germany

http://www.maartenbuis.nl
---------------------------------

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/