Re: st: file cannot be opened in loop appending many datasets

[Stefan Bernhard <stefanbernhard88@gmail.com>]

Sergiy, thanks so much for your little program!
I had the same problem with looping and appending over several
thousand, and this speeds up my process by some 1000%, as I had added
a -sleep 500- after each loop to reduce the likelihood of failure
(which it still did several times after a few hundred loops).

On a sidenote, isn't it somehow possible to tell Stata to use the RAM
for -local tempfile-s instead of the HDD? It seems to be a wasteful
process to write to the mechanic disc tens of thousands of times
during such loops as the here mentioned, instead of using the
non-mechanical RAM.

On a second sidenote, I tried to add a -noisily- to your -display-
line in fileop, but the quitely of my own loop suppresses the output.
What are the mechanics behind this? Can I somehow make noisily go back
up through all 'layers' of quietly? That being without adding
-noisily- before the -fileop- in the respective -save- operation,
because I don't want to read the saving info.

i.e. during:

qui {
foreach something {
fileop save "test.dta", replace
}
}

I want to know when fileop did its job and read the line "Encountered
error `ecode'.`emsg'"


Best Regards, Stefan Bernhard,

2013/7/24 Sergiy Radyakin <serjradyakin@gmail.com>:
> See if -fileop- can help you.
>
> radyakin.org/statalist/2013072401/fileop.htm
>
> Best, Sergiy Radyakin
>
>
>
>
> On Wed, Jul 24, 2013 at 11:15 AM, adrien bouguen
> <abouguen@povertyactionlab.org> wrote:
>> Dear all,
>>
>> I am trying to append a large number of datasets using 2 loops and I am
>> getting a strange error. After a while, the loop stops with a r(603) error
>> saying that the data cannot be opened whereas the data is there and was not
>> opened before.
>>
>> The error seems to be affected datasets randomly : it is never the same
>> dataset that can't be opened and it is never the same loop number (here
>> count) that poses the problem.
>>
>> I suspect a problem with the memory but I have tried this command in
>> different computers without success. See below for the code.
>>
>> Any ideas?
>>
>> Thanks so much for your help
>>
>>
>> clear
>> set mem 2g
>> set virtual on
>> global dta "Q:/numerique/dta"
>> global text "Q:/numerique/text"
>>
>> clear
>> set mem 1000
>> set virtual on
>> local files : dir "$dta/enfants" files "*"
>> dis `files'
>> global count=0
>> foreach file in `files' {
>> local z=subinstr("`file'",".dta","",.)
>> use `"$var/`z'_v1.dta"', clear
>> global count=$count+1
>> forval j=3(2)31 {
>> dis "`j'"
>> cap append using `"$var/`z'_v`j'.dta"'
>> if _rc==0 {
>> local y=`j'+1
>> drop if v1=="" & v`j'==""
>> replace v2=v`y' if v1==""
>> replace v1=v`j' if v1==""
>> drop v`j' v`y'
>> }
>> }
>> save `"$col/`z'_colonne.dta"', replace
>> clear
>> }
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/