Re: st: collapse is too memory demanding

[Ada Ma <heu034@googlemail.com>]

You don't need the reference groups for the dummy variables. e.g. you
don't need both m_total and f_total, you can drop one of them.

After that if it still doesn't work you might want to try using
broader regions / broader occupation groups.

HTH.

Ada


On Mon, Jul 26, 2010 at 7:43 PM, Oliver Jones
<ojones@wiwi.uni-bielefeld.de> wrote:
> Hi everybody,
> this is my first posting and I hope to ask a good question...
> How much additional free memory do I need to perfom a -collapse-?
>
> Problem setting:
> I have a dataset containing information about ~20 million people living in
> ~180 different regions and working in ~330 different jobs. The information
> is given by ~70 zero/one dummy variables, like male[yes/no], female[yes/no],
> age20-25[yes/no], ...
> When I try to collapse it like this I get the error that I need more free
> memory
>
> **********
> * begin excerpt code
> *
> * m_total is a dummy variable taking the value 1 if the person is male
> * f_total is a dummy variable taking the value 1 if the person is female
> *
> collapse (sum) m_total f_total ...(68 more dummy variables), by(aoaa
> beruford) fast
> *
> *
> .
> .
> no room to add more variables because of width
>    An attempt was made to add a variable that would have increased the
> memory required to store an observation beyond what is currently possible.
>  You have the
>    following alternatives:
>
>     1.  Store existing variables more efficiently; see help compress.
>
>     2.  Drop some variables or observations; see help drop.  (Think of
> Stata's data area as the area of a rectangle; Stata can trade off width and
> length.)
>
>     3.  Increase the amount of memory allocated to the data area using the
> set memory command; see help memory.
> r(902);
> *
> *
> memory
>
> .         memory
>                                                  bytes
> --------------------------------------------------------------------
> Details of set memory usage
>    overhead (pointers)                     159,493,976        8.45%
>    data                                  1,455,382,531       77.11%
>                                        ----------------------------
>    data + overhead                       1,614,876,507       85.56%
>    free                                    272,560,293       14.44%
>                                        ----------------------------
>    Total allocated                       1,887,436,800      100.00%
> --------------------------------------------------------------------
> Other memory usage
>    set maxvar usage                          2,041,738
>    set matsize usage                         1,315,200
>    programs, saved results, etc.                37,424
>                                        ---------------
>    Total                                     3,394,362
> -------------------------------------------------------
> Grand total                               1,890,831,162
>
> *
> *
> * end code excerpt
> **********
>
>
>
> I am grateful for any help.
>
> Kind regards
> Oliver
>
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>



-- 
Ada Ma
Research Fellow
Health Economics Research Unit
University of Aberdeen, UK.
http://www.abdn.ac.uk/heru/
Tel: +44 (0) 1224 555189
Fax: +44 (0) 1224 550926

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/