Re: st: using Stata to detect interviewer fraud

[Stas Kolenikov <skolenik@gmail.com>]

On Fri, Apr 30, 2010 at 10:16 PM, Michelson, Ethan <emichels@indiana.edu> wrote:
> I'd be deeply grateful for help writing a more efficient, more parsimonious .do file to help detect interviewer fraud. After completing a survey of 2,500 households, I discovered that a few interviewers copied each others' questionnaires. I decided to write some code that calculates the proportion of all nonmissing questionnaire items that are identical across every other questionnaire. Although my .do file accomplishes this task, I strongly suspect I'm making Stata do tons of unnecessary work. It takes Stata about 12 hours to process 505 questionnaires (from a single survey site, since I can rule out the possibility that interviewers conspired across different survey sites).....

I imagine the tasks like comparing the data between rows are better
accomplished by -cluster- routines. They aren't lightning fast,
either, but I believe they are better optimized for speed than your
code is, but still have the same functionality of determining how
"close" the two sets of responses are. -cluster- works great if you
have either all continuous or all discrete data; if you have a mix, or
if you have missing data, the choices are more limited, and you'd need
to come up with more inventive ways of computing the differences
between the completed questionnaires.

That'll make a terrific Stata conference talk, you know :)). I don't
remember seeing anything like that at the US meetings.

-- 
Stas Kolenikov, also found at http://stas.kolenikov.name
Small print: I use this email account for mailing lists only.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/