Hello Mr. Peng,
a number of developers have released "Memory Defragmenters". Most of
them claim to be able to defragment memory/unload unnecessary DLLs,
etc. Do you have any experience with any of these programs? Is this a
potential solution?
Alternatively, since 32-bit machines are going to be around for a
while, why not allow Stata to work with non-continuous memory
allocation? I can understand the benefits of having the whole dataset
in memory, but why should it be continuous? The whole dataset may
still be hold in memory, but there will be no problem with memory
fragmentation. Since the issue seems to be with Windows, other
developers have to face the same problems and there might be already a
suitable solution for that (working with photos and video can eat GBs
in no time, and even MS Messenger seems to be using more DLLs than
Stata does <--showdep). In the worst-case scenario, the whole dataset
can be represented as a set of pointers and data containers occupying
as little as several bytes each.
As for unpredictibility of the memory allocation here is a quote from MS:
"/REBASE This option sets the base addresses for the specified files.
EDITBIN assigns new base addresses in a contiguous address space
according to the size of each file rounded up to the nearest 64 KB.
For details about base addresses, see the Base Address (/BASE) linker
option."
Does this have any impact on the actual addresses of the DLLs?
How about the /PAE and /3GB switches?
Some ramdisk developers claim using 3GB on WinXP32-bit:
http://www.ei-europe.com/Default.aspx?tabid=318#s
Having 600M-900M memory available for data is not cool anymore with
most workstations running Stata having already 2GB+ of physical memory
installed.
I guess the FAQ needs an update on how to effectively overcome
limitations of OS/Stata/Hardware/etc while working with large data,
rather than explaining why it is not possible or another version is
required.
Best regards,
Sergiy Radyakin
On 9/12/07, Hua Peng <[email protected]> wrote:
> Arne Mastekaasa <[email protected]> asks:
> "Under Windows XP 32 bit I am able to allocate 900m of memory under Stata 9,
> but only 600m under Stata 10 (-set memory- command). Anybody else experienced
> this? Can anything be done? (other than changing to 64 bit system)."
>
> Arne may find the following FAQ providing useful information.
>
> http://www.stata.com/support/faqs/win/winmemory.html
>
> Quoting from the above FAQ,
>
> "When a typical application loads, there are usually several
> libraries (or DLLs) that are loaded as well. These libraries are
> usually loaded into the 2.1-GB space on the upper end, but not in
> any deterministic order. Microsoft has assured us that there is no
> way to prevent these libraries from loading into arbitrary
> addresses, thus fragmenting the available space. When Stata tries
> to load a dataset, it requests from Windows the largest contiguous
> space in the 2.1-GB range. Depending on where Windows loaded the
> initial libraries, this may be 1.8 GB, 1.3 GB, or even less. You
> may be surprised to find that a 1.4-GB dataset loaded fine one time
> but failed to load later. This is simply an unfortunate side effect
> of Windows memory management."
>
> Since Stata 10 uses more Microsoft libraries than Stata 9, the virtual
> memory space in Stata 10 is more fragmented than in Stata 9. Hence
> less memory is available.
>
> If Arne often needs more than 1GB memory to work with large dataset, a
> 64-bit Stata on a 64-bit operating system is the solution.
>
> On the other hand, if Arne only needs around 900MB memory, he may try
> to apply the Microsoft hotfix in the FAQ first. If this does not fix
> the problem, he may consider moving from 32-bit XP to 32-bit Vista
> since the virtual memory management in Vista is much better than in
> XP.
>
> --
> -------------------------------------------------------------------
> Hua Peng, Ph.D.
> Senior Software Engineer VOICE: 1-979-696-4600
> StataCorp LP TOLL : 1-800-782-8272
> 4905 Lakeway Drive FAX : 1-979-696-4601
> College Station, TX 77845 EMAIL: [email protected]
> -------------------------------------------------------------------
>
> *
> * For searches and help try:
> * http://www.stata.com/support/faqs/res/findit.html
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
>
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
