> ----- Original Message -----
> From: "Tim R. Sass" <[email protected]>
> Date: Wednesday, July 14, 2004 12:17 pm
> Subject: st: Memory Usage
>
> > I may be crazy, but I am trying to estimate a fixed effects model
> > with a
> > handful of explanatory variables plus 3,100 explicit dummies on a
> > two-year
> > panel data set containing 1.7 million observations (about 850,000
> > fixed
> > effects). I am using a Sun workstation with 8GB of RAM.
> >
>
I don't the problem is memory.
To form X'X the computer is going to do a lot of arithmetic. For each of
1.7 million observations at least 3,100 items have to be multiplied times
each of 3,100 other items, and added to X'X. That is about 17,000 trillion
multiplies and adds.
Then the resulting matrix (which is about 40 megabytes of data) must be
inverted, which will require an awfull lot of elementary row operations -
too many to do in a lifetime I expect. Doesn't the cost of the matrix
inversion grow as the cube of the number of variables in the regression?
Of course depending on the values of the dummies the matrix may be very
sparse, but that would help only if Stata could take advantage of
sparseness, which I don't recall hearing that it could, and with areg, I
think the matrix would not be sparse.
Daniel Feenberg
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
