Re: st: Op. sys. refuses to provide memory - a cautionary tale

[Eric Booth <ebooth@ppri.tamu.edu>]

<>
On Aug 20, 2010, at 4:07 PM, Tony wrote:
>  Too much RAM will slow it down.

Stata will certainly slow down if you set and use more memory in Stata than is physically available on your machine because you start using virtual memory, but beyond this limit Stata won't slow down as you add or allocate more RAM.  
That is, it will take the same time to run a do-file on a 1G dataset whether you allocate 2G, 8G, or 20G of memory to Stata; however it will get excruciatingly slow if you set your memory to 500m and  use a 1G dataset.


> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M. wrote:

>> The purpose is real world speed, so has anyone actually noticed if going
>> up to 24GB RAM, solid state drive expedited your jobs


I haven't tested the idea of getting a SSD drive, but I think the speed advantage would be evident mainly when you were opening(reading) or saving(writing) a large dataset since your using the data in memory the rest of the time.  I do want to try out installing a SSD drive for working with data that is larger than my physical RAM and requires me to use virtual memory to work with it (I've maxed out my physical RAM with 8G sticks in each of the slots).  I've read about moving your page/swap file to a SSD which should speed things up when working in virtual memory (but since SSDs wear out faster with more read/writes, this might be a concern).  Also, you could move the location of the tempfiles that Stata creates to that path by setting your OS system temp file location (see:  http://www.stata.com/statalist/archive/2009-05/msg00416.html).  Maybe someone here has tried working with SSD and large datasets ?

Again, more RAM is always better IMO--but it only speeds you up in the sense that it prevents you from using page file.  There are also speeds associated with RAM (mine is 1066 DDR3), but I don't know much differences in memory speed matters.


> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M. wrote:
>> Does any listserv member think that I should go from 6 to 12 cores?
>> 
>>  Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>  Dual Six Core Processor,X5680,3.33GHz,12M,6.4GT/s

Depends on what you are doing.  If you've got a time intensive procedure that you're running on your 6 core machine, try running it with your -set processors- at 1, 2, 4, and 6 and see what kind of speed increase you observe, e.g.:

*****!
timer clear 
forval n = 1(2)6 {
clear all
set mem 32g
set processors `n'
timer on `n'
	<your command goes here>
timer off `n'
	}
timer list
*****!

~ Eric


__
Eric A. Booth
Public Policy Research Institute
Texas A&M University
ebooth@ppri.tamu.edu
Office: +979.845.6754


> 
> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M.
> <Benjamin.Craig@moffitt.org> wrote:
>> Thanks Nick, I have learned that to truly take advantage of the latest
>> version of Stata, 64-bits and 4 or more cores is required. To be a bit
>> more specific, lets assume I am using 6-core Stata MP on Windows 7
>> Professional, 64-bit for computationally intensive simulation analyses.
>> 
>> Does any listserv member think that I should go from 6 to 12 cores?
>> 
>>  Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>  Dual Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>> 
>> Is it worthwhile to upgrade from RAM and Hard drive? For example,
>> 
>>  12GB DDR3 ECC SDRAM Memory,1333MHz,6X2GB
>> 
>>  1TB SATA 3.0Gb/s,7200 RPM HardDrive with 32MB DataBurst Cache
>> 
>> The purpose is real world speed, so has anyone actually noticed if going
>> up to 24GB RAM, solid state drive expedited your jobs? In theory, it
>> should, but I am hoping that someone has purchase a computer recently to
>> test this hypothesis.
>> 
>> Cheers,
>> Ben
>> 
>> 
>> Benjamin M. Craig, PhD
>> 
>> Assistant Member, Health Outcomes & Behavior, Moffitt Cancer Center
>> 
>> Associate Professor, Economics, University of South Florida
>> 
>> 12902 Magnolia Dr, MRC-CANCONT, Tampa, FL 33612-9416
>> 
>> Phone (813) 745-6710; Fax (813) 745-6525
>> 
>> -----Original Message-----
>> From: owner-statalist@hsphsun2.harvard.edu
>> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
>> Sent: Friday, August 20, 2010 5:00 AM
>> To: 'statalist@hsphsun2.harvard.edu'
>> Subject: RE: st: Op. sys. refuses to provide memory - a cautionary tale
>> 
>> I think this partially answers itself in that I don't think that it can
>> fairly be expected that a company website is a proper place for a
>> company, in this case StataCorp, to offer opinions about anything that
>> is currently controversial.
>> 
>> That said, Benjamin's question is obviously practical and a fair one for
>> members of this list to venture opinions and comment from experience. In
>> addition, presumably people other than econometricians are not excluded.
>> 
>> 
>> Nick
>> n.j.cox@durham.ac.uk
>> 
>> Craig, Benjamin M.
>> 
>> To clarify, the website answers essential questions (which systems are
>> supported?) and provides some advice and education. I need to know more
>> details on current controversies relating to multiple core, 64-bit and
>> drive speeds in the real world. In this sense, it is a bit incomplete to
>> say that more core, more bits, and faster drives are preferable. As was
>> previously post, I had thought that a 32-bit dual core desktop was good
>> enough for my needs, and was woefully wrong. Others seem to follow...
>> 
>> http://www.stata.com/statalist/archive/2010-07/msg01337.html
>> 
>> If a listserv member has tried STATA MP on multiple machines, I would
>> like to know what worked best so that I can buy one. There is an obvious
>> caveat: speed depends on the task. However, I would counter that some
>> data are better than none. For example, I recently bootstrapped a ML
>> with 1000 iteration and inequality constraints. It took 4 weeks. Do you
>> think your machine can do better? Personally, I do not use stata for
>> database management, and doubt that many econometricians do. If someone
>> has a good analytics machine, and he/she thinks that it works well, I'd
>> like to know its components. Maybe a consensus will emerge. Maybe not.
>> 
>> Martin Weiss
>> 
>> In which respect is the Stata website "incomplete"? There is advice at
>> http://www.stata.com/products/opsysmp.html, and how is the website
>> supposed to give more detailed advice? It does not know your specific
>> setup, hence the reluctance to go into greater depth...
>> 
>> Craig, Benjamin M.
>> 
>> Okay, I give up... I need a new machine. Due to institutional policies,
>> I need to buy a Dell. Otherwise, I would very much like any advice on
>> this purchase. My best guess is a 64-bit 8-core desktop for a 6-core
>> version of Stata MP. I don't need a rocket, just a racecar.
>> 
>> If you have any specifications that you would like to share with me,
>> please send them directly or post them on the listserv for others
>> looking to upgrade. I have read the Stata website, which seems
>> incomplete.


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/