Re: st: xt: unit-specific trends

[Nick Cox <njcoxstata@gmail.com>]

As I understand it:

Work with -if- is really dumb. Stata tests every observation. Even if
you say -if _n < 7- there is no intelligence saying "Oh, we're done
here" once you get to observation 7. How could there be? In that
example and some others working with -in- is a much, much better idea.
However, you're working with -if `touse'- here. One alternative is

preserve
keep if `touse'
...
restore

but you still have to use -if-. It could be that

replace `touse' = -`touse'
sort `touse'
qui count if `touse'
local last = r(N)

and then doing things

...  in 1/`last'

helps.

As for other stuff, I think most of the answers would be "It depends".

On what MP does and doesn't do, see http://www.stata.com/statamp/statamp.pdf

Over time, the Mata content of Stata is going up, but many of the
basic operations are done using compiled C.

I think Stata tech-support would need much more information than this
on your data, your code and your machine before they could make many
useful comments.

More positively, you could use -set rmsg on- to see what is going most slowly.

Nick

2012/4/19 László Sándor <sandorl@gmail.com>:
> Quick comments on this:
>
> I forgot to flag that the residual variable need to exist beforehand
> for -genbump- below, this is only replacing values of it.
>
> More importantly: The operation is still far, far from linear in the
> number of individuals (N in the panel — T is fixed). I could again
> finish a 1% subsample in around 10 minutes or so, but my bold attempt
> at 10% overnight still only finished 4 out of the 8 variables to be
> transformed this way in 10 or 11 hours.
>
> Maybe caching and memory is an issue here, but if anybody (StataCorp?)
> had a comment on this otherwise, that would be helpful.
>
> Maybe firing up _regress and _predict all the time is very costly? Or
> the marksample is not fast enough with the by option? (Does the code
> know that once it finished with seven consecutive rows there is
> nothing to check further below "whether" `touse' is 1 anywhere else? I
> guessed byable commands produce efficient subscripting for some
> underlying Mata code…) Or even the byable command does not use MP
> resources efficiently? (Still, even remaining serial, the speed-up
> could be much closer to linear, no?)
>
> I thought individual-specific trends are almost as trendy nowadays as
> fixed-effects — I wonder if they could be done much faster.
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/