RE: st: Reading large data sets in Stata

["Martin Weiss" <martin.weiss1@gmx.de>]

<>
" That might have appeared in a different posting that I missed."

I mentioned it twice, once implicitly in
http://www.stata.com/statalist/archive/2010-02/msg01006.html, then
explicitly in http://www.stata.com/statalist/archive/2010-02/msg01014.html.
Still, it deserves the prominence that it is getting in this thread.


HTH
Martin


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Michael Norman
Mitchell
Sent: Montag, 22. Februar 2010 21:43
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: Reading large data sets in Stata

To add to Kit's excellent answer, I would add the link to the 
"interactive dataset calculator" he referred to...

http://www.stata.com/support/faqs/data/datacalc.cgi

That might have appeared in a different posting that I missed.

Michael N. Mitchell
See the Stata tidbit of the week at...
http://www.MichaelNormanMitchell.com
Visit me on Facebook at...
http://www.facebook.com/MichaelNormanMitchell

On 2010-02-22 12.19 PM, Christopher Baum wrote:
> <>
> Stas said
>
> That's 13Gb of data, right? If you really want to put everything into 
> memory, then you would probably need a computer with 24Gb of RAM. I 
> don't really know if you can buy anything like that in the desktop 
> format, and what kind of OS you would need to look at, although I am 
> sure there are clusters with much larger memory capacities. If you 
> only need subsets of that data set, then you could use <list of the 
> variables that you REALLY need> if <subsetting to the conditions you 
> REALLY want to analyze> using <this huge data set name> That way, you 
> may have a data set of a more realistic 2Gb size that you can work 
> with on a 4Gb RAM machine.
>
> That's not necessarily 13 Gb of data. Using the interactive calculator 
> on the FAQ, if you assume all 37 variables can be held in 4 bytes 
> each, it's under 7 Gb. If on average they only need 3 bytes each, it's 
> under 6 Gb. Stat/Transfer can optimize the dataset as it converts it 
> to Stata format. Stas' suggestions are well taken, but one more is 
> important--if any of these variables are 0/1 indicators, or integers 
> taking on values 1..5, etc. they need not chew up nearly as much 
> memory. I don't know if you can get it down to a 2 Gb size, though. To 
> use more than 2 Gb, you need a 64-bit machine (almost all machines are 
> these days), and Stata 11 will automatically install the 64-bit 
> version on such a machine.
>
> Kit Baum   |   Boston College Economics and DIW Berlin   |   
> http://ideas.repec.org/e/pba1.html
> An Introduction to Stata Programming   |   
> http://www.stata-press.com/books/isp.html
> An Introduction to Modern Econometrics Using Stata   |   
> http://www.stata-press.com/books/imeus.html
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/