Re: st: Weights

["Michael Blasnik" <michael.blasnik@verizon.net>]

...
I find this response somewhat frustrating to read -- apparently resolving the 
problem isn't very important to you but important enough to ask people's advice 
on Statalist.  If the Stata file is twice the size of the ASCII CSV file then 
you are very likely storing some numbers using bigger variable types than you 
need.  Given the number of lines and overall size of the file, it seems quite 
unlikely that the problem is related to SPSS being able to compress strings.  In 
addition, there are very few numeric fields that most people use that actually 
need double precision storage.  If you know so little about the data contained 
in the variables that you can't make storage type decisions then I would guess 
that you don't need those variables and so your problem can be solved by just 
dropping the variables you know nothing about.

M Blasnik

----- Original Message ----- 
From: "Martin Weiss" <martin.weiss@uni-tuebingen.de>
To: <statalist@hsphsun2.harvard.edu>
Sent: Thursday, May 01, 2008 10:02 AM
Subject: RE: st: Weights



> Thanks for all replies,
>
>
> yet the issue is not at all resolved. I had the file broken along columns
> overnight and -compress-ed. As there are slightly more than 600 columns, I
> went for 7 package @ approx. 90 each. Later, I tried to -merge- them but the
> result is almost exactly the same compared to splitting along the rows and
> -append-ing. Setting -mem- to 3G, which is perfectly feasible on 64 bit
> machines, I managed to -merge- four out of seven. On the fifth run thru the
> -forv- loop, Stata complained about lack of -mem-. Adding up the size of the
> 7 -compress-ed files yields the size of 5.5 G mentioned in the initial post
> in this thread. -Compress-ing did not help much, neither did the -makesmall-
> prog which was posted in this thread. With regard to -recast, force- that
> sets off alarm bells for me as it might destroy valuable information (which
> I cannot check for every var as there are over 600 of them...). I do
> actually like the peace of mind that comes with -compress-...
> Now, I cannot possibly post anything on the data as they are confidential. I
> do notice, though, that SPSS 16.0 manages them @ half the file size and @
> amazing speed. So far, I have only tried descriptives, but as things stand,
> I may not need much more with this dataset. Not willing to go thru the
> frustration of learning the ropes in another package, I went 64 bit
> precisely to avoid the quandary I am now in.
> As things stand, and in contrast to one of the posts earlier, there is
> something magic about the way data are stored and processed in different
> packages...
>
> Martin Weiss
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/