Re: st: xtreg: double format and memory issues

[Enrico Cavale <tolacahc@gmail.com>]

Xtreg, at least in the way I use it and I reported here, takes out the
means for the individual_by_year, but introduce 720 dummy variables
for day FE. That suggests that we need something like 550000
observations*720 day FE* 8 bytes = 3,186 megabytes + 550,000
observations*13 variables*8 bytes .= 6.6 megabyte. More or less 3200
megabytes, but I can allocate only 2900 mb.

I use 8 bytes because I consider that the function regress and xtreg
generates doubles and not floats. Indeed, if they were floats, and
then 4 bytes, I would have half the memory requirements and I would be
able to estimate my model.

Enrico

2011/3/24 Daniel Feenberg <feenberg@nber.org>:
>
>
> On Thu, 24 Mar 2011, Nick Cox wrote:
>
>> My guess is that this is not a route that will bring you much joy if
>> only because -regress- and -xtregress-, elaborate though they are, are
>> just wrappers for internal code that you cannot tune.
>>
>> Sorry, but I think you do need more memory or a smaller problem.
>>
>> Nick
>>
>> On Thu, Mar 24, 2011 at 5:45 PM, Enrico Cavale <tolacahc@gmail.com> wrote:
>>
>>> I'm writing to statalist because I have a memory issue. I need to
>>> estimate a two-way fixed effects model with the constant. In
>>> particular I have 12 regressors, 550000 observations, 3066
>>> individual_by_year fixed effects and 720 daily fixed effects. In stata
>>> it looks like:
>>>
>>> xtreg y x1 x2 .... x12 _Iday_*,  fe i(individual_by_year)
>>> cluster(individual)
>>>
>>> If I'm right I would need something like 3200 mb to run that command,
>>> but I can allocate only 2900 mb. Nevertheless, I noticed that the
>
> I am interested in why the command should take so much memory.  Doesn't
> xtreg take out means, so that the cross product matrix is only 13x13? That
> would suggest that 550,000 observations*15 variables*4 bytes/item equals 33
> megabytes should be sufficient to estimate the model. But I expect I am
> missing something.
>
> Daniel Feenberg

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/