-findit limits- leads to a help page discussing the dataset limitations of
the various flavours of Stata:
Small Intercooled Stata/SE
# obs about 1,000 2,147,483,647 2,147,483,647
# var 99 2,047 32,767
dataset width 200 24,564 393,192
It may not be that you're out of memory, rather out of Stata's ability to
manipulate the data. I'd suspect the dataset width is where you're falling
afoul, as neither the variable count nor observation size seems to be close
to the limits of SE (assuming that's what you're using). I assume that
figure is measured in bytes.
-----Original Message-----
From: Tim R. Sass [mailto:[email protected]]
Sent: Wednesday, July 14, 2004 10:18 AM
To: [email protected]
Subject: st: Memory Usage
I may be crazy, but I am trying to estimate a fixed effects model with a
handful of explanatory variables plus 3,100 explicit dummies on a two-year
panel data set containing 1.7 million observations (about 850,000 fixed
effects). I am using a Sun workstation with 8GB of RAM.
I initially set the maxvar, memory and matsize to yield the following:
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 6000 max. variables allowed 1.935M
set memory 6500M max. data space 6,500.000M
set matsize 5000 max. RHS vars in models 191.154M
-----------
6,693.090M
After making some variable transformations I compress the data and then run
areg:
. xi: areg nrtrgain nschools chgschl
> t2001 tgrde_04 tgrde_06 tgrde_07 tgrde_08 tgrde_09 tgrde_10
> rpeat_04 rpeat_05 rpeat_06 rpeat_07 rpeat_08 rpeat_09 rpeat_10
> i.instid,
> absorb(student) robust;
All of the dummy variables get created, but I run out of memory before the
regression estimates are computed:
i.instid _Iinstid_1-36994 (naturally coded; _Iinstid_1 omitted)
no room to add more variables due to width
I still have about 1.3 GB of memory free (see below), though I guess that
is not enough to do the matrix inversions necessary to compute the
regression estimates. Is there a way to figure out the memory that would
be required to solve this problem? I have some money budgeted to buy a
64-bit machine with 16GB RAM in the future, though I'm not sure even such a
machine could do this. Any suggestions?
Tim
. memory
bytes
--------------------------------------------------------------------
Details of set memory usage
overhead (pointers) 6,974,664 0.10%
data 5,499,522,564 80.69%
----------------------------
data + overhead 5,506,497,228 80.79%
free 1,309,246,772 19.21%
----------------------------
Total allocated 6,815,744,000 100.00%
--------------------------------------------------------------------
Other memory usage
set maxvar usage 2,029,457
set matsize usage 200,440,000
programs, saved results, etc. 120,605
---------------
Total 202,590,062
-------------------------------------------------------
Grand total 7,018,334,062
Tim R. Sass
Professor Voice: (850)644-7087
Department of Economics Fax: (850)644-4535
Florida State University E-mail: [email protected]
Tallahassee, FL 32306-2180 Internet: http://garnet.acns.fsu.edu/~tsass
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
