Re: st: Stata corrupting data?

[Arnold Kester <arnold.kester@stat.unimaas.nl>]

Deborah Garvey wrote:
> Hi, all.
>
> I use Intercooled Stata 7.0 Win 98/95/NT version (born 6/11/02) on a
> Dell OptiPlex GX110 with 512 MB RAM running Win 98.  My computer has
> about 7 GB disk space available for virtual memory.  (Obviously not the
> fastest machine on earth.)
>
> A worrisome event occurred yesterday while reading in a 1990 Census 5%
> PUMS CA sample abstracted from the IPUMS web site.  The data set is 312m
> in size with N = 1.46 m and 84 vars.
>
> I set memory at 375 m. After I read in the data to STATA, ran
> descriptive stats, and saved the data, I then did a quick cross-tab of 2
> variables that should've yielded a 2x2 matrix.  Instead, values were
> changed for a couple of observations, and I ended up with a 3x3 table. 
> I not so calmly exited STATA, restarted my computer and checked the
> data.  They seem to be fine.
>
> The data were seriously corrupted when I initially attempted to read in
> 1990 and 2000 5% PUMS abstracts for CA simultaneously.  I verified with
> IPUMS that the problem was on my end, and not with their source data.
>
> Has this happened to anyone else before?   Is it just a matter of being
> more generous in allocating virtual memory?
I had this last year (using stata 7) with a much smaller dataset, about 
800k. I did a series of merge operations that required saving temporary 
datafiles on a server-disk over the network, and that sometimes resulted 
in corruption. I put several -assert- commands in my .do files to stop 
processing when corruption occurred.
When I moved the temp files to a local disk the problem disappeared. It 
could have been a network error. I'm not working on that project 
anymore, so I cannot reproduce the fault now.

Regards,
Arnold Kester

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/