Re: st: file cannot be opened in loop appending many datasets

[adrien bouguen <abouguen@povertyactionlab.org>]

Thanks Sergiy.

Though, your program seems not to be working with a "capture" as it is
the case in my loop.

 I have resolved the problem by adding a "sleep 2000" after saving the
data. But it is true Stefan that it slows down the process a lot.

To understand what's going on from a non-IT specialist person, the
error is caused by the fact that data are stored in a hard drive while
it is manipulated in the RAM. Since the RAM is way faster than the
hard drive it messes things up. Is that correct?

Thanks

2013/7/25 Stefan Bernhard <stefanbernhard88@gmail.com>:
> Sergiy, thanks so much for your little program!
> I had the same problem with looping and appending over several
> thousand, and this speeds up my process by some 1000%, as I had added
> a -sleep 500- after each loop to reduce the likelihood of failure
> (which it still did several times after a few hundred loops).
>
> On a sidenote, isn't it somehow possible to tell Stata to use the RAM
> for -local tempfile-s instead of the HDD? It seems to be a wasteful
> process to write to the mechanic disc tens of thousands of times
> during such loops as the here mentioned, instead of using the
> non-mechanical RAM.
>
> On a second sidenote, I tried to add a -noisily- to your -display-
> line in fileop, but the quitely of my own loop suppresses the output.
> What are the mechanics behind this? Can I somehow make noisily go back
> up through all 'layers' of quietly? That being without adding
> -noisily- before the -fileop- in the respective -save- operation,
> because I don't want to read the saving info.
>
> i.e. during:
>
> qui {
> foreach something {
> fileop save "test.dta", replace
> }
> }
>
> I want to know when fileop did its job and read the line "Encountered
> error `ecode'.`emsg'"
>
>
> Best Regards, Stefan Bernhard,
>
> 2013/7/24 Sergiy Radyakin <serjradyakin@gmail.com>:
>> See if -fileop- can help you.
>>
>> radyakin.org/statalist/2013072401/fileop.htm
>>
>> Best, Sergiy Radyakin
>>
>>
>>
>>
>> On Wed, Jul 24, 2013 at 11:15 AM, adrien bouguen
>> <abouguen@povertyactionlab.org> wrote:
>>> Dear all,
>>>
>>> I am trying to append a large number of datasets using 2 loops and I am
>>> getting a strange error. After a while, the loop stops with a r(603) error
>>> saying that the data cannot be opened whereas the data is there and was not
>>> opened before.
>>>
>>> The error seems to be affected datasets randomly : it is never the same
>>> dataset that can't be opened and it is never the same loop number (here
>>> count) that poses the problem.
>>>
>>> I suspect a problem with the memory but I have tried this command in
>>> different computers without success. See below for the code.
>>>
>>> Any ideas?
>>>
>>> Thanks so much for your help
>>>
>>>
>>> clear
>>> set mem 2g
>>> set virtual on
>>> global dta "Q:/numerique/dta"
>>> global text "Q:/numerique/text"
>>>
>>> clear
>>> set mem 1000
>>> set virtual on
>>> local files : dir "$dta/enfants" files "*"
>>> dis `files'
>>> global count=0
>>> foreach file in `files' {
>>> local z=subinstr("`file'",".dta","",.)
>>> use `"$var/`z'_v1.dta"', clear
>>> global count=$count+1
>>> forval j=3(2)31 {
>>> dis "`j'"
>>> cap append using `"$var/`z'_v`j'.dta"'
>>> if _rc==0 {
>>> local y=`j'+1
>>> drop if v1=="" & v`j'==""
>>> replace v2=v`y' if v1==""
>>> replace v1=v`j' if v1==""
>>> drop v`j' v`y'
>>> }
>>> }
>>> save `"$col/`z'_colonne.dta"', replace
>>> clear
>>> }
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/



-- 
Adrien Bouguen
Ecole d'économie de Paris - J-PAL
66 bis avenue Jean Moulin
75014 Paris
tel: +33 (0)1 71 19 40 83
port: +33 (0)6 69 34 12 02

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/