st: Speed issues / which disk for tempfiles

[Ernest Berkhout <ernestb@seo.fee.uva.nl>]

Hi all,

I like to share some thought mainly on speeding up Stata, in particular the 
reshape-command. Presently I run a do-file in which i ask Stata 8.2 to do 
some reshaping, including the reshape of some str244-variables. The 
original file is 350.000 obs, the reshaped file somewhere about 6000 obs. 
Of course this takes a very long time (about 10 minutes) and I'm looking 
for tips to shorten this type of tasks.

I already discovered that quite a gain can be made if i split the data set 
in half, then do a reshape on both subsets, and then merge them together.

Also I recognized that during the reshape, the CPU usage on a 
Windows2000-system is only 30-50%, and the memory usage is only 25% of max. 
Of course 'set virtual' is turned off. At the same time, reading and 
writing to disc is quite heavy. From this I deduced that the best way to 
speed up the process is to speed up the reading/writing to disk, as this 
seems to be the bottleneck.

This raises the question of how that can be done best, and why Stata does 
not use the whole off the available memory. Is it possible to specify on 
which disk or directory Stata writes its temporary files, for instance?

Ernest Berkhout
Stichting voor Economisch Onderzoek
Universiteit van Amsterdam

Kamer 3.08
Roetersstraat 29
1018 WB Amsterdam

tel.:+ 31 20 525 1630
fax:+ 31 20 525 1686
http://www.seo.nl
=================================
"To an economist, real life is a special case"
=================================

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/