I am amazed with the level of help that you get from here. Let me
start off expressing my gratitude for this.
To answer some of Nick's points:
1. The loop consists of repeated -drop-ping of observations not
desired, working with the remaining subset and then a -restore- of
the original. It is difficult to say in general what is most
efficient and what most elegant but for a situation like that below
I'd normally just add an extra condition excluding the observations
not wanted, rather than repeatedly doing major surgery on the
dataset. However, others could equally point out that applying -if-
on a very large dataset can be time-consuming.
Answer #1: I have been using ?preserve- & -restore, preserve- in loops
because of system limitations. I have a dataset as below
Contains data from us_data_ret.dta
obs: 17,516,468
vars: 5 10 Nov 2009 07:02
size: 525,494,040 (56.6% of memory free)
------------------------------------------------------------
storage display value
variable name type format label variable label
-------------------------------------------------------------
id int %8.0g ID
dscd str6 %9s DSCD
date float %td
year int %8.0g
totalReturn double %10.0g
-----------------------------------------------------------
Sorted by: id date
that I join with annual market data as below:
Contains data from C:\Users\Joachim\AppData\Local\Temp\ST_02000005.tmp
obs: 1,461 Total returns for
value-weighted ave S/B & H/L BM portf and their diffs. See Fam
vars: 5 9 Dec 2009 23:08
size: 46,752 (99.9% of memory free)
--------------------------------------------------------------------------------------------------------
storage display value
variable name type format label variable label
--------------------------------------------------------------------------------------------------------
date float %td
SMBtr float %9.0g Small-Minus-Big Total
Returns. See Fama & French (1993).
HMLtr float %9.0g High-Minus-Low Total
Returns. See Fama & French (1993).
riskFreereturn float %9.0g US T-Bill 3 month, middle rate
marketReturn double %9.0g Frank Russel 3000, Total
Return Index
-------------------------------------------------------------------------------------------------------
Sorted by:
The first dataset above is in "long format" and has 10826 unique ids
(US firms). The date variable ranges from 03Jan1978 to 30dec2008 with
weekly stock market data.
Then I wish to do a Fama & French 3F CAPM calculation per id and then,
based on the 3F CAPM I wish to calculate the expected annual return
for each firm-year when each firm is traded on the stock market. Now I
know I may be using too much lingo, but this boils down to an OLS
regression per id provided that enough data is available (for me that
implies three years of weekly data).
Since I do not have enough computing power/memory (I am using a
Windows 32-bit system) to add more variables into the joined dataset I
have resorted use the ?preserve- & -restore, preserve- method to drop
all but a single id and then do the necessary operations on that
subset. Then I save the data into a tempfile. Later (after 6-7 hours),
when the loop is done I use ?append- to build the dataset anew (having
dropped unnecessary data).
Any suggestions to more elegant methods are greatly appreciated.
2. If only the minimum and maximum are needed from a -summarize- it
is best just to use a -meanonly- option. (The name -meanonly- is
misleading, as I've had occasion to remark before.)
I will have the ?meanonly- option in mind. Did not know about it. :)
3. Code like
local `minDate' = r(min)
<stuff> if <stuff> date >= ``minDate''
looks legal but odd. You are probably using more levels of macros
than you need. It's hard to tell because the code isn't completely
self-contained (that's not a criticism; it wasn't necessary for your
question).
I am indeed confused with the Stata usage of ` ' and have been having
problems with this. minDate is a ?tempname- so if I just write
`minDate' I don't get the value, just the tempname. Doing it as
``minDate'' provides me with the value. But I am sure there are
smarter ways to do this. . .
4. Code in which you loop over the contents of a local macro and
change that macro within the loop can be tricky. Watch out!
My computer at the university is running a test this evening with the
revised code and I was intuitively also worried to change ?panelVar-
inside the loop which it guides. I have done it as Martin proposed and
added a ?adjustedPanelVar- that I modify. I need it for a second loop
where I append the dataset once the first loop is done.
5. The -if- condition in
summarize totalReturn if totalReturn != .
is unnecessary as -summarize- always ignores missings.
That is always good to know.
6. To get minimum and maximum dates in a panel, no looping is necessary as
egen mindate = min(date), by(id)
egen maxdate = max(date), by(id)
will do it. Similarly it looks as if your main problem does not need
any looping either, as it should yield to -egen- operations. Look
at -egen, count()- in particular.
Yes I have used ?min()-elsewhere but I cannot easily add more
variables to the complete database due to memory limitations. Memory
is a luxury to me. . .
7. More generally, it is not always positive to know too many other
languages if they lead you to seek a Stata equivalent of other code
when there's a Stataish way to do it without any real programming.
I completely agree. Coming from SAS I am used to ?proc sql- and SAS'
-proc iml-. Had I just known about STATA 1998 this would not have been
a problem. Now I have to "delearn" the SAS version of sql while
learning STATA and that is like a baby learning to walk. I try my best
but keep bruising myself. . . :)
I will respond to Martin's second post tomorrow by posting the code
and an an excerpt of the data (beware that it looks much like crochet
since I have mainly been trying to get the code to run without exiting
with errors).
/Joachim
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
