Re: st: how can i make my loop run faster?

[Austin Nichols <austinnichols@gmail.com>]

Stefano Rossi <sr525@cornell.edu>:
You can sort your data and use -in- instead of -if- to reduce
processing time considerably.  You can generate regression coefs and
predictions without ever firing up -regress-, probably using lag
operators or a wide date structure, which can also speed things up.

See e.g.
http://www.stata.com/statalist/archive/2008-10/msg00136.html
and for some related material:
http://www.stata.com/statalist/archive/2011-01/msg00662.html
http://www.stata.com/statalist/archive/2008-04/msg00967.html

On Tue, Sep 20, 2011 at 1:24 AM, Partho Sarkar
<partho.ss+lists@gmail.com> wrote:
> Hello Stefanlo
>
> I am not quite sure I understand what you are proposing, but I suppose
> this ought to be do-able,  though I can't offhand think of any direct
> way to do this.  I think this would also be an extremely cumbersome
> way to do this, and almost certainly  even slower than any of the
> others we have talked about.  However, I had thought of another
> possible approach:
>
> Since you have a "short" panel- "only" 200 periods, but many more
> firms, I would think dividing up your sample period into, say, 10
> (non-overlapping) subsets, and doing a statsby regression for each
> would give you the results for all (or selected subset of) firms
> within each time sub-period.  You would still have to
> combine(merge/append) the results.
>
> But why not try the rolling loop as suggested in the thread I cited,
> first splitting the firms into manageable subsets (which I think you
> would have to do in any case, if you want to run this routine- unless
> you have access to a super-computer!)  ?
>
> By the way, I am curious to know what exactly the source of your data is!
>
> Hope this helps
>
> Partho
>
>
> On Tue, Sep 20, 2011 at 10:20 AM, Stefano Rossi <sr525@cornell.edu> wrote:
>> Dear Partho,
>>
>> many thanks for this, which is very useful.  I can see how "rolling" works, and I can see how it can generate efficiency gains, but I agree the whole procedure may still be quite slow and require splitting the sample into subsets to get a faster procedure in some way.
>>
>> I am currently considering a different path, namely generating a cross-section of observations by firm-period, whereby each firm-period unit contains 12 observations, from -1 to -12 (I would also have a separate data by +1 to +12).  This procedure would effectively produce a dataset 12 times larger than my current one.  This procedure would get around the "rolling" issue, and would allow me to use the "statsby" (or equivalent) command without worrying of the length of the estimation sample, with potentially large efficiency improvements (i.e., no "ifs").
>>
>> Provided my intuition is correct, my one concern here is how to create such dataset, which is 12 times bigger than the current one.  Is there a built-in Stata command that allows to do this efficiently?
>>
>> Many thanks for your support.
>>
>> Kind regards,
>>
>> Stefano
>>
>>
>> ________________________________________
>> From: owner-statalist@hsphsun2.harvard.edu [owner-statalist@hsphsun2.harvard.edu] On Behalf Of Partho Sarkar [partho.ss+lists@gmail.com]
>> Sent: Tuesday, September 20, 2011 12:33 AM
>> To: statalist@hsphsun2.harvard.edu
>> Subject: Re: st: how can i make my loop run faster?
>>
>> I guess Stefano might have solved his problem by now, but just to
>> complete this, here is a post by  Brian R. Landy from an older thread
>> which gives the complete code for -rolling-, including merging the
>> results files.
>>
>> http://www.stata.com/statalist/archive/2009-09/msg01239.html
>>
>> The thread also points out the speed problems with rolling for panel data.
>>
>> P.Sarkar

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/