st: Memory Usage

["Tim R. Sass" <tsass@coss.fsu.edu>]

I may be crazy, but I am trying to estimate a fixed effects model with a 
handful of explanatory variables plus 3,100 explicit dummies on a two-year 
panel data set containing 1.7 million observations (about 850,000 fixed 
effects).  I am using a Sun workstation with 8GB of RAM.

I initially set the maxvar, memory and matsize to yield the following:

                    current                                 memory usage
    settable          value     description                 (1M = 1024k)
    --------------------------------------------------------------------
    set maxvar         6000     max. variables allowed           1.935M
    set memory         6500M    max. data space              6,500.000M
    set matsize        5000     max. RHS vars in models        191.154M
                                                            -----------
                                                             6,693.090M

After making some variable transformations I compress the data and then run 
areg:

. xi: areg  nrtrgain nschools chgschl
>           t2001 tgrde_04 tgrde_06 tgrde_07 tgrde_08 tgrde_09 tgrde_10
>           rpeat_04 rpeat_05 rpeat_06 rpeat_07 rpeat_08 rpeat_09 rpeat_10
>           i.instid,
>           absorb(student) robust;

All of the dummy variables get created, but I run out of memory before the 
regression estimates are computed:

i.instid          _Iinstid_1-36994    (naturally coded; _Iinstid_1 omitted)
no room to add more variables due to width

I still have about 1.3 GB of memory free (see below), though I guess that 
is not enough to do the matrix inversions necessary to compute the 
regression estimates.  Is there a way to figure out the memory that would 
be required to solve this problem?  I have some money budgeted to buy a 
64-bit machine with 16GB RAM in the future, though I'm not sure even such a 
machine could do this.  Any suggestions?

Tim


. memory
                                                  bytes
--------------------------------------------------------------------
Details of set memory usage
    overhead (pointers)                       6,974,664        0.10%
    data                                  5,499,522,564       80.69%
                                        ----------------------------
    data + overhead                       5,506,497,228       80.79%
    free                                  1,309,246,772       19.21%
                                        ----------------------------
    Total allocated                       6,815,744,000      100.00%
--------------------------------------------------------------------
Other memory usage
    set maxvar usage                          2,029,457
    set matsize usage                       200,440,000
    programs, saved results, etc.               120,605
                                        ---------------
    Total                                   202,590,062
-------------------------------------------------------
Grand total                               7,018,334,062




Tim R. Sass
Professor                               Voice:   (850)644-7087
Department of Economics         Fax:      (850)644-4535
Florida State University                E-mail:   tsass@coss.fsu.edu
Tallahassee, FL  32306-2180     Internet: http://garnet.acns.fsu.edu/~tsass


*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/