Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: st: Op. sys. refuses to provide memory - a cautionary tale
From
"Jay Tuthill" <[email protected]>
To
<[email protected]>
Subject
RE: st: Op. sys. refuses to provide memory - a cautionary tale
Date
Tue, 31 Aug 2010 11:43:53 -0400
Thanks for this link. It's going to take me a while to digest it.
Regards...Jay
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Sergiy Radyakin
Sent: Tuesday, August 31, 2010 10:56 AM
To: [email protected]
Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
2010/8/30 Jay Tuthill <[email protected]>:
> I saw where you showed that Stata addressed 50GB of memory. How can you tell how much of that memory is in RAM versus what is allocated to the harddisk?
obviously not all of it is in the RAM. The tools and methods are described here:
http://www.ibm.com/developerworks/library/j-memusage/
Best, Sergiy
>
> Thanks...Jay
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of Sergiy Radyakin
> Sent: Friday, August 27, 2010 12:31 PM
> To: [email protected]
> Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
>
> 2010/8/26 Jay Tuthill <[email protected]>:
>> First, I did not wish to imply I am building a workstation with 192GB of RAM. That is a little too expensive for me, but I have settled on 32GB of RAM. I have not picked an operating system yet. The following website may be of use to those who are looking for the memory limits of various operating systems. It is from Kingston Technologies who have an excellent Memory Assessor under the Memory Tools tab. This page is for Windows 7; other operating systems are available from the assessor page.
>>
>> http://www.kingston.com/tools/assessor/win7.asp
>>
>> As to how much memory Stata will address once I have this up and running, I will not know for some weeks. It is not such a simple question as this machine runs two processors and each processor has 16GB of memory. The processors each have 4 cores for a total of 8 cores and I am licensed for Stata 4MP. Hence, I don't know if Stata will take at most 16GB, or will make use of more. Perhaps someone who has already built a large system can weigh in.
>
> With a proper OS Stata will be able to claim more than your installed
> 32GB (thanks to the virtual memory in Windows).
> Each processor does not have its dedicated 16GB. (though on the board
> you will have two banks of memory).
> http://img706.imageshack.us/img706/2160/memp.png
> on a machine with 24G physical mem, as shown Stata has claimed 50G.
>
> Hope this helps, Sergiy
>
>
>
>
>>
>> Regards...Jay
>>
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On Behalf Of Sergiy Radyakin
>> Sent: Tuesday, August 24, 2010 2:46 PM
>> To: [email protected]
>> Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
>>
>> Martin, this is very vogue. What is the "total amount of memory on
>> your machine"?
>> Is this the amount of hardware sticks of DIMMs (or whatever
>> technology) that you've
>> paid your dollars (euros, pounds) for? or is it the amount of memory
>> that Windows
>> can see and make use of?
>>
>> The answer should probably be "the least of ...." followed by some
>> parameters of the
>> hardware and software.
>>
>> Most importantly, even some 64-bit flavors of Windows face the 2GB
>> physical memory
>> limit (e.g. Win7 Starter Edition), which means that if you are running
>> such a system,
>> installing more hardware memory will not do you any good. 16GB is a
>> limit for several
>> 64-bit flavors of Windows: e.g. Windows Vista Home Premium.
>>
>> Let me put a link to the Microsoft's website, as this is very volatile
>> information:
>> http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx
>>
>> The "availability" is also technology-dependent. Refer for example to
>> the Stata 1.0
>> manual. It says, in particular: "When STATA is started, the program
>> examines your PC,
>> determines how much memory is available, and then lays claim to all of
>> it. Thus the
>> size of the largest data set that you can use depends upon the
>> hardware configuration
>> of your PC". If Stata 1.0 was available now, and you ran it in Windows
>> 7, would it claim
>> all the GBs installed and available? No. However the statement I
>> quoted from the manual
>> was true, in that sense Stata will make use of all the "memory that is
>> made available to it",
>> which may differ however large from "memory that is put into the
>> computer box". In the
>> case of Stata 1.0 it would mean that it will honestly make use of 640k
>> out of perhaps 8GB
>> or whatever you have in your desktop. [anyone still having Stata 1.0
>> can give it a shot, and
>> let me use an opportunity and ask here if anyone still has a copy of
>> Stata for DOS? or for
>> that matter, what is the earliest version of Stata that is currently
>> in operation?]
>>
>> 2Benjamin: I understand your frastration, however, if you are in
>> charge of getting your own
>> hardware and software, configuring it all and making it work together,
>> find some time to
>> learn about the potential problems, incompatibilities, limits and
>> adjust your expectations
>> accordingly. Working with large datasets with Stata is possible, and
>> all the necessary
>> information is available and made public, and nobody intentionally
>> hides it from you. If you
>> don't have time to check this out, it is always possible to delegate
>> such tasks to others,
>> or simply to ask around for ready solutions. Perhaps you only need to
>> upgrade the Windows
>> since the hardware itself has been 64-bit for years now.
>>
>> "Does any listserv member think that I should go from 6 to 12 cores?"
>> Yes! Why not? What do we know about your expected returns from the
>> project, availability
>> of funds? Do you even face a budget constraint? If not, go for the
>> 32-cores (current max for
>> Stata/MP11)...
>>
>> 2Eric: Quote: "however it will get excruciatingly slow if you set your
>> memory to 500m and
>> use a 1G dataset." I don't think it is going to be slow, it just not
>> going to work. Stata will say
>> "not enough memory". I'd rather agree to the following: "if you
>> install 512mb of physical
>> memory into the computer, but allocate 1GB in Stata it will slow down"
>> (and note: regardless (!)
>> of the size of the dataset).
>>
>> Best,
>> Sergiy Radyakin
>>
>>
>> 2010/8/24 Martin Weiss <[email protected]>:
>>>
>>> <>
>>>
>>>
>>> " but what is the final word? How much memory can a 64bit system handle?"
>>>
>>>
>>>
>>>
>>> " You are limited only by the total amount of memory on your machine", see
>>> http://www.stata.com/products/64bitintro.html
>>>
>>>
>>>
>>>
>>> HTH
>>> Martin
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: [email protected]
>>> [mailto:[email protected]] Im Auftrag von Kiss Sándor
>>> Csanád
>>> Gesendet: Dienstag, 24. August 2010 11:53
>>> An: [email protected]
>>> Betreff: RE: st: Op. sys. refuses to provide memory - a cautionary tale
>>>
>>> Sorry guys, I read all this, and also this
>>>
>>> http://www.stata.com/statalist/archive/2010-01/msg00409.html
>>>
>>> but what is the final word? How much memory can a 64bit system handle?
>>>
>>> Thanks
>>>
>>>
>>>
>>> Kiss Sándor Csanád
>>> Elemző közgazdász / Economist
>>> Office of the Fiscal Council, Hungary
>>> 1055 Budapest, Honvéd utca 20.
>>> T: (+36 1) 510 3025
>>> Mobil: (+36 30) 703 1024
>>> Fax: (+36 1) 510 3099
>>> Web: www.mkkt.hu
>>>
>>>
>>> -----Original Message-----
>>> From: [email protected]
>>> [mailto:[email protected]] On Behalf Of Jeph Herrin
>>> Sent: Monday, August 23, 2010 7:30 PM
>>> To: [email protected]
>>> Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
>>>
>>> I took the plunge on Windows 7 64bit and have been very
>>> satisfied with it relative to XP 64bit, which is what that machine
>>> was running. Though with only 16gb RAM, I haven't really put
>>> it to the test you envision with 192gb.
>>>
>>> For what it's worth, I shop my PCs from a custom builder that
>>> caters to engineers and CAD/CAM types; they were dead set against
>>> selling Vista but said that Win7 had so far proved itself for
>>> their customers and they were recommending it for those who
>>> needed not-Linux.
>>>
>>> cheers,
>>> Jeph
>>>
>>>
>>> On 8/23/2010 12:45 PM, Jay Tuthill wrote:
>>>> Hi All,
>>>>
>>>> I'm also in the middle of upgrading my hardware as my current 32 bit
>>>> computers will not let me run very efficiently the datasets I want. I'm
>>>> considering the new Intel workstation system SC5650SCWSR which offers
>>>> dual Xeon processors and up to 192GB of RAM. Am curious if anyone has
>>>> compared the Windows 7 64bit versus Window Server 2008 R2 64bit
>>>> operating systems. (I have built my own computers for several years and
>>>> have a wide latitude in how I configure them.)
>>>>
>>>> Thanks...Jay
>>>>
>>>> -----Original Message-----
>>>> From: [email protected]
>>>> [mailto:[email protected]] On Behalf Of Eric Booth
>>>> Sent: Saturday, August 21, 2010 5:24 PM
>>>> To:<[email protected]>
>>>> Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
>>>>
>>>> <>
>>>>
>>>> On Aug 21, 2010, at 4:19 PM, Eric Booth wrote:
>>>>
>>>>> but beyond this limit Stata won't slow down as you add or allocate
>>>> more RAM.
>>>>
>>>> Clarification:
>>>> That should say, "but UP TO (or before you reach this memory limit)
>>>> Stata won't slow down as you add or allocate more RAM..."
>>>>
>>>> On Aug 21, 2010, at 4:19 PM, Eric Booth wrote:
>>>>
>>>>> <>
>>>>> On Aug 20, 2010, at 4:07 PM, Tony wrote:
>>>>>> Too much RAM will slow it down.
>>>>>
>>>>> Stata will certainly slow down if you set and use more memory in Stata
>>>> than is physically available on your machine because you start using
>>>> virtual memory, but beyond this limit Stata won't slow down as you add
>>>> or allocate more RAM.
>>>>> That is, it will take the same time to run a do-file on a 1G dataset
>>>> whether you allocate 2G, 8G, or 20G of memory to Stata; however it will
>>>> get excruciatingly slow if you set your memory to 500m and use a 1G
>>>> dataset.
>>>>>
>>>>>
>>>>>> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M. wrote:
>>>>>
>>>>>>> The purpose is real world speed, so has anyone actually noticed if
>>>> going
>>>>>>> up to 24GB RAM, solid state drive expedited your jobs
>>>>>
>>>>>
>>>>> I haven't tested the idea of getting a SSD drive, but I think the
>>>> speed advantage would be evident mainly when you were opening(reading)
>>>> or saving(writing) a large dataset since your using the data in memory
>>>> the rest of the time. I do want to try out installing a SSD drive for
>>>> working with data that is larger than my physical RAM and requires me to
>>>> use virtual memory to work with it (I've maxed out my physical RAM with
>>>> 8G sticks in each of the slots). I've read about moving your page/swap
>>>> file to a SSD which should speed things up when working in virtual
>>>> memory (but since SSDs wear out faster with more read/writes, this might
>>>> be a concern). Also, you could move the location of the tempfiles that
>>>> Stata creates to that path by setting your OS system temp file location
>>>> (see: http://www.stata.com/statalist/archive/2009-05/msg00416.html).
>>>> Maybe someone here has tried working with SSD and large datasets ?
>>>>>
>>>>> Again, more RAM is always better IMO--but it only speeds you up in the
>>>> sense that it prevents you from using page file. There are also speeds
>>>> associated with RAM (mine is 1066 DDR3), but I don't know much
>>>> differences in memory speed matters.
>>>>>
>>>>>
>>>>>> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M. wrote:
>>>>>>> Does any listserv member think that I should go from 6 to 12 cores?
>>>>>>>
>>>>>>> Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>>>> Dual Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>>
>>>>> Depends on what you are doing. If you've got a time intensive
>>>> procedure that you're running on your 6 core machine, try running it
>>>> with your -set processors- at 1, 2, 4, and 6 and see what kind of speed
>>>> increase you observe, e.g.:
>>>>>
>>>>> *****!
>>>>> timer clear
>>>>> forval n = 1(2)6 {
>>>>> clear all
>>>>> set mem 32g
>>>>> set processors `n'
>>>>> timer on `n'
>>>>> <your command goes here>
>>>>> timer off `n'
>>>>> }
>>>>> timer list
>>>>> *****!
>>>>>
>>>>> ~ Eric
>>>>>
>>>>>
>>>>> __
>>>>> Eric A. Booth
>>>>> Public Policy Research Institute
>>>>> Texas A&M University
>>>>> [email protected]
>>>>> Office: +979.845.6754
>>>>>
>>>>>
>>>>>>
>>>>>> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M.
>>>>>> <[email protected]> wrote:
>>>>>>> Thanks Nick, I have learned that to truly take advantage of the
>>>> latest
>>>>>>> version of Stata, 64-bits and 4 or more cores is required. To be a
>>>> bit
>>>>>>> more specific, lets assume I am using 6-core Stata MP on Windows 7
>>>>>>> Professional, 64-bit for computationally intensive simulation
>>>> analyses.
>>>>>>>
>>>>>>> Does any listserv member think that I should go from 6 to 12 cores?
>>>>>>>
>>>>>>> Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>>>> Dual Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>>>>
>>>>>>> Is it worthwhile to upgrade from RAM and Hard drive? For example,
>>>>>>>
>>>>>>> 12GB DDR3 ECC SDRAM Memory,1333MHz,6X2GB
>>>>>>>
>>>>>>> 1TB SATA 3.0Gb/s,7200 RPM HardDrive with 32MB DataBurst Cache
>>>>>>>
>>>>>>> The purpose is real world speed, so has anyone actually noticed if
>>>> going
>>>>>>> up to 24GB RAM, solid state drive expedited your jobs? In theory, it
>>>>>>> should, but I am hoping that someone has purchase a computer
>>>> recently to
>>>>>>> test this hypothesis.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Ben
>>>>>>>
>>>>>>>
>>>>>>> Benjamin M. Craig, PhD
>>>>>>>
>>>>>>> Assistant Member, Health Outcomes& Behavior, Moffitt Cancer Center
>>>>>>>
>>>>>>> Associate Professor, Economics, University of South Florida
>>>>>>>
>>>>>>> 12902 Magnolia Dr, MRC-CANCONT, Tampa, FL 33612-9416
>>>>>>>
>>>>>>> Phone (813) 745-6710; Fax (813) 745-6525
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: [email protected]
>>>>>>> [mailto:[email protected]] On Behalf Of Nick Cox
>>>>>>> Sent: Friday, August 20, 2010 5:00 AM
>>>>>>> To: '[email protected]'
>>>>>>> Subject: RE: st: Op. sys. refuses to provide memory - a cautionary
>>>> tale
>>>>>>>
>>>>>>> I think this partially answers itself in that I don't think that it
>>>> can
>>>>>>> fairly be expected that a company website is a proper place for a
>>>>>>> company, in this case StataCorp, to offer opinions about anything
>>>> that
>>>>>>> is currently controversial.
>>>>>>>
>>>>>>> That said, Benjamin's question is obviously practical and a fair one
>>>> for
>>>>>>> members of this list to venture opinions and comment from
>>>> experience. In
>>>>>>> addition, presumably people other than econometricians are not
>>>> excluded.
>>>>>>>
>>>>>>>
>>>>>>> Nick
>>>>>>> [email protected]
>>>>>>>
>>>>>>> Craig, Benjamin M.
>>>>>>>
>>>>>>> To clarify, the website answers essential questions (which systems
>>>> are
>>>>>>> supported?) and provides some advice and education. I need to know
>>>> more
>>>>>>> details on current controversies relating to multiple core, 64-bit
>>>> and
>>>>>>> drive speeds in the real world. In this sense, it is a bit
>>>> incomplete to
>>>>>>> say that more core, more bits, and faster drives are preferable. As
>>>> was
>>>>>>> previously post, I had thought that a 32-bit dual core desktop was
>>>> good
>>>>>>> enough for my needs, and was woefully wrong. Others seem to
>>>> follow...
>>>>>>>
>>>>>>> http://www.stata.com/statalist/archive/2010-07/msg01337.html
>>>>>>>
>>>>>>> If a listserv member has tried STATA MP on multiple machines, I
>>>> would
>>>>>>> like to know what worked best so that I can buy one. There is an
>>>> obvious
>>>>>>> caveat: speed depends on the task. However, I would counter that
>>>> some
>>>>>>> data are better than none. For example, I recently bootstrapped a ML
>>>>>>> with 1000 iteration and inequality constraints. It took 4 weeks. Do
>>>> you
>>>>>>> think your machine can do better? Personally, I do not use stata for
>>>>>>> database management, and doubt that many econometricians do. If
>>>> someone
>>>>>>> has a good analytics machine, and he/she thinks that it works well,
>>>> I'd
>>>>>>> like to know its components. Maybe a consensus will emerge. Maybe
>>>> not.
>>>>>>>
>>>>>>> Martin Weiss
>>>>>>>
>>>>>>> In which respect is the Stata website "incomplete"? There is advice
>>>> at
>>>>>>> http://www.stata.com/products/opsysmp.html, and how is the website
>>>>>>> supposed to give more detailed advice? It does not know your
>>>> specific
>>>>>>> setup, hence the reluctance to go into greater depth...
>>>>>>>
>>>>>>> Craig, Benjamin M.
>>>>>>>
>>>>>>> Okay, I give up... I need a new machine. Due to institutional
>>>> policies,
>>>>>>> I need to buy a Dell. Otherwise, I would very much like any advice
>>>> on
>>>>>>> this purchase. My best guess is a 64-bit 8-core desktop for a 6-core
>>>>>>> version of Stata MP. I don't need a rocket, just a racecar.
>>>>>>>
>>>>>>> If you have any specifications that you would like to share with me,
>>>>>>> please send them directly or post them on the listserv for others
>>>>>>> looking to upgrade. I have read the Stata website, which seems
>>>>>>> incomplete.
>>>>>
>>>>>
>>>>> *
>>>>> * For searches and help try:
>>>>> * http://www.stata.com/help.cgi?search
>>>>> * http://www.stata.com/support/statalist/faq
>>>>> * http://www.ats.ucla.edu/stat/stata/
>>>>
>>>>
>>>> *
>>>> * For searches and help try:
>>>> * http://www.stata.com/help.cgi?search
>>>> * http://www.stata.com/support/statalist/faq
>>>> * http://www.ats.ucla.edu/stat/stata/
>>>>
>>>> *
>>>> * For searches and help try:
>>>> * http://www.stata.com/help.cgi?search
>>>> * http://www.stata.com/support/statalist/faq
>>>> * http://www.ats.ucla.edu/stat/stata/
>>>>
>>> *
>>> * For searches and help try:
>>> * http://www.stata.com/help.cgi?search
>>> * http://www.stata.com/support/statalist/faq
>>> * http://www.ats.ucla.edu/stat/stata/
>>>
>>> *
>>> * For searches and help try:
>>> * http://www.stata.com/help.cgi?search
>>> * http://www.stata.com/support/statalist/faq
>>> * http://www.ats.ucla.edu/stat/stata/
>>>
>>>
>>> *
>>> * For searches and help try:
>>> * http://www.stata.com/help.cgi?search
>>> * http://www.stata.com/support/statalist/faq
>>> * http://www.ats.ucla.edu/stat/stata/
>>>
>>
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/statalist/faq
>> * http://www.ats.ucla.edu/stat/stata/
>>
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/statalist/faq
>> * http://www.ats.ucla.edu/stat/stata/
>>
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
>
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
