Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: Need help with working with multiple files
From
Nick Cox <[email protected]>
To
"[email protected]" <[email protected]>
Subject
Re: st: Need help with working with multiple files
Date
Tue, 28 May 2013 14:30:51 +0100
has a inner loop using the same macro name for loop element as the
outer loop.
The villain in the piece is
foreach x of varlist price mpg rep78 trunk{
which has the side-effect of zapping local macro x once it has finished.
The rule is that a local macro defined by a loop is undefined once
that loop finishes. It is, doubly, local.
Thus `x' is defined <here> but not <there> in something like
foreach x of local X {
* here `x' is OK
foreach x of local Y {
}
* there `x' has been undefined
}
Remedy: Don't do that. Use a different name, e.g.
foreach y of local Y
Nick
[email protected]
On 28 May 2013 14:03, Michael Stewart <[email protected]> wrote:
> Dear Statalist user ,
> I am having problem with saving files after damamanagement tasks using
> a locals in foreach loop and facing difficulty.(As I try to work with
> three datasets, the output is only one dataset).I have tried many
> options, sought help of statlist but able to correct the problem
> partially but there is a bug somewhere. I am here with posting the
> code and stata output with "traceon".May be any one could please help
> me.
>
> MY CODE / DO FILE IS AS FOLLOWS
> local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
> local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
> local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
> local all ds1 ds2 ds3
> foreach x in `all'{
> use ``x'',clear
> di "``x''"
> include "D:\_Dummy\multiple\4\mpg.do"
> di "``x''"
> include "D:\_Dummy\multiple\5\sum.do"
> save ``x''_new.dta,replace
> }
>
> mpg DO FILE IS AS FOLLOWS ( NO FOREACH LOOP IN THIS DOFILE)
> gen mean=sum(mpg)
>
> sum DOFILE IS AS FOLLOWS (FOREACH LOOP PRESENT IN THIS DOFILE)
> foreach x of varlist price mpg rep78 trunk{
> gen x_`x'=`x'==3
> }
>
>
> PROBLEM: instead of generating three new datasets , it is generating
> one dataset _new.
>
> MY STATA OUTPUT IS AS FOLLOWS(WITH TRACE ON AND DEBUGGNG STEP di
> ``x''); LOOKS LIKE AFTER EXECUTUNG THE SECOND DOFILE (WITH FOREACH
> LOOP) di ``x'' shows nothing!!!!!
>
> ****************************************************************************************************************************
> local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
>
> . local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
>
> . local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
>
> . local all ds1 ds2 ds3
>
> . foreach x in `all'{
> 2. use ``x'',clear
> 3. di "``x''"
> 4. include "D:\_Dummy\multiple\4\mpg.do"
> 5. di "``x''"
> 6. include "D:\_Dummy\multiple\5\sum.do"
> 7. save ``x''_new.dta,replace
> 8. }
> - foreach x in `all'{
> = foreach x in ds1 ds2 ds3{
> - use ``x'',clear
> = use D:\_Dummy\multiple\1\auto_1.dta,clear
> (1978 Automobile Data)
> - di "``x''"
> = di "D:\_Dummy\multiple\1\auto_1.dta"
> D:\_Dummy\multiple\1\auto_1.dta
> - include "D:\_Dummy\multiple\4\mpg.do"
>
> . gen mean=sum(mpg)
>
> .
> . - di "``x''"
> = di "D:\_Dummy\multiple\1\auto_1.dta"
> D:\_Dummy\multiple\1\auto_1.dta
> - include "D:\_Dummy\multiple\5\sum.do"
>
> . gen xx= sum(weight)
>
> . foreach x of varlist price mpg rep78 trunk{
> 2. gen x_`x'=`x'==3
> 3. }
> - foreach x of varlist price mpg rep78 trunk{
> - gen x_`x'=`x'==3
> = gen x_price=price==3
> - }
> - gen x_`x'=`x'==3
> = gen x_mpg=mpg==3
> - }
> - gen x_`x'=`x'==3
> = gen x_rep78=rep78==3
> - }
> - gen x_`x'=`x'==3
> = gen x_trunk=trunk==3
> - }
>
> .
> . - save ``x''_new.dta,replace
> = save _new.dta,replace
> file _new.dta saved
> - }
> - use ``x'',clear
> = use D:\_Dummy\multiple\2\auto_2.dta,clear
> (1978 Automobile Data)
> - di "``x''"
> = di "D:\_Dummy\multiple\2\auto_2.dta"
> D:\_Dummy\multiple\2\auto_2.dta
> - include "D:\_Dummy\multiple\4\mpg.do"
>
> . gen mean=sum(mpg)
>
> .
> . - di "``x''"
> = di "D:\_Dummy\multiple\2\auto_2.dta"
> D:\_Dummy\multiple\2\auto_2.dta
> - include "D:\_Dummy\multiple\5\sum.do"
>
> . gen xx= sum(weight)
>
> . foreach x of varlist price mpg rep78 trunk{
> 2. gen x_`x'=`x'==3
> 3. }
> - foreach x of varlist price mpg rep78 trunk{
> - gen x_`x'=`x'==3
> = gen x_price=price==3
> - }
> - gen x_`x'=`x'==3
> = gen x_mpg=mpg==3
> - }
> - gen x_`x'=`x'==3
> = gen x_rep78=rep78==3
> - }
> - gen x_`x'=`x'==3
> = gen x_trunk=trunk==3
> - }
>
> .
> . - save ``x''_new.dta,replace
> = save _new.dta,replace
> file _new.dta saved
> - }
> - use ``x'',clear
> = use D:\_Dummy\multiple\3\auto_3.dta,clear
> (1978 Automobile Data)
> - di "``x''"
> = di "D:\_Dummy\multiple\3\auto_3.dta"
> D:\_Dummy\multiple\3\auto_3.dta
> - include "D:\_Dummy\multiple\4\mpg.do"
>
> . gen mean=sum(mpg)
>
> .
> . - di "``x''"
> = di "D:\_Dummy\multiple\3\auto_3.dta"
> D:\_Dummy\multiple\3\auto_3.dta
> - include "D:\_Dummy\multiple\5\sum.do"
>
> . gen xx= sum(weight)
>
> . foreach x of varlist price mpg rep78 trunk{
> 2. gen x_`x'=`x'==3
> 3. }
> - foreach x of varlist price mpg rep78 trunk{
> - gen x_`x'=`x'==3
> = gen x_price=price==3
> - }
> - gen x_`x'=`x'==3
> = gen x_mpg=mpg==3
> - }
> - gen x_`x'=`x'==3
> = gen x_rep78=rep78==3
> - }
> - gen x_`x'=`x'==3
> = gen x_trunk=trunk==3
> - }
>
> .
> . - save ``x''_new.dta,replace
> = save _new.dta,replace
> file _new.dta saved
> - }
>
> .
> end of do-file
> ****************************************************************************************************************************
>
>
> So as you have noticed a, after executing dofile "sum", the naleof
> the file is being removed and di "" shows nothing.
>
> Can anyone of the users help fix the bug.
>
> Thank you
> Mike
>
>
>
>
> On Tue, May 28, 2013 at 7:37 AM, Nick Cox <[email protected]> wrote:
>> It is best to think that you are addressing Statalist rather than
>> think that one person is necessarily going to answer. Assume that
>> people at the other end are just as capricious as you might be and get
>> bored, need to meet partners, attend to the day job, or write their
>> novel instead. On the whole the list is quite good at picking up
>> answerable questions but we don't have a buddy system.
>>
>> From your example it seems likeliest that one of the extra loops you
>> are including is somehow zapping the local macro x. I don't have
>> another guess. Inner loops are not in principle fatal to outer loops
>> -- otherwise programming would hardly be possible -- but it seems that
>> your code is stomping on itself.
>>
>> Your code is longer and more complicated than you care to show and you
>> can't win easily here. Post too much code or too little; either can be
>> wrong in terms of not getting a useful reply.
>>
>> But I would check that you are not using the same macro name for
>> different loops.
>>
>> Also, you can add debugging steps
>>
>> di "`mymacro'"
>>
>> to get Stata to show what it thinks -mymacro- is (or is not).
>>
>> Example where nested loops are fine.
>>
>> . foreach x in a b c {
>> 2. di "first time `x'"
>> 3. foreach y in 1 2 3 {
>> 4. di "`y'"
>> 5. }
>> 6. di "second time `x'"
>> 7. }
>> first time a
>> 1
>> 2
>> 3
>> second time a
>> first time b
>> 1
>> 2
>> 3
>> second time b
>> first time c
>> 1
>> 2
>> 3
>> second time c
>> Nick
>> [email protected]
>>
>>
>> On 28 May 2013 11:24, Michael Stewart <[email protected]> wrote:
>>> HI Nick
>>> Sorry to keep bothering you as I wanted to clarify the last few
>>> sentences of your reply "save the file within
>>> the loop _or_ otherwise use the name you want directly."
>>>
>>> Here in my situation, the code would be
>>>
>>> clear
>>> local ds1 c\data\top/dataset
>>> local ds2 c\data\date/news/dataset
>>> ...
>>> ...
>>> ..
>>> local ds13 c\data\data/datedataset-
>>> local all ds1 ....ds13
>>> foreach x in `all'{
>>> use ``x'',clear
>>> datamagement tasks / contains foreaach loop
>>> datamagement tasks / contains foreaach loop
>>> datamagement tasks / contains foreaach loop
>>>
>>> save ?????
>>> }
>>>
>>> Here while saving the dataset , when I use save ``x''_new,replace-->
>>> this saves the dataset as _new and we get one dataset instead of the
>>> 13 dataset that we started with
>>>
>>> According to your recommendation "otherwise use the name you want
>>> directly", how can I fit 13 names here in the loop .
>>> Am I missing some important concept here?? Please help me with the
>>> problem and understand the concept.
>>>
>>> Thanks
>>> MIke
>>>
>>>
>>> On Tue, May 28, 2013 at 3:40 AM, Nick Cox <[email protected]> wrote:
>>>> If you refer to a macro that doesn't exist Stata doesn't regard that
>>>> as an error: it just substitutes nothing, or equivalently ignores the
>>>> reference.
>>>>
>>>> Nested macro references don't affect this principle and indeed must be
>>>> consistent with it. So if local macro x does not exist as a string to
>>>> be used neither do any of
>>>>
>>>> `x'
>>>> ``x''
>>>> ```x''''
>>>>
>>>> and so forth. You could say that they define empty strings if you
>>>> like, but the effect is the same: a non-existent string and an empty
>>>> string look identical.
>>>>
>>>> From what you say that is precisely what you did. It was the -foreach-
>>>> loop that defined x as what the loop used -- and (an extra principle
>>>> here) that macro is not defined outside the loop, and indeed is doubly
>>>> local, to the loop as well as in the usual sense.
>>>>
>>>> The user manual [U] contains a basic introduction to macros in
>>>>
>>>> [U] Chapter 18 . . . . . . . . . . . . . . . . . . . . Programming Stata
>>>> (help program, macro, quotes, syntax, examplehelpfile)
>>>>
>>>> From your earlier posts it seems that you need to save the file within
>>>> the loop _or_ otherwise use the name you want directly.
>>>>
>>>> Nick
>>>> [email protected]
>>>>
>>>> On 28 May 2013 06:00, Michael Stewart <[email protected]> wrote:
>>>>> Dear Steve and Nick ,
>>>>> I am having problem with saving the files
>>>>> I am using ``x'' and the dofile runs through multiple dofiles like a charm
>>>>> However, after completetion of the datamanagement tasks, the each
>>>>> file is being stored as "_new" in the stata working directory.
>>>>> And the interesting part is that this happens only if the individual
>>>>> tasks have foreach loop in it. If the individual tasks dont have
>>>>> foreach loop in it, then the files are saved in respective folders
>>>>> i.e., save ``x''_new, replace --> works
>>>>> I am not sure is there is a way to circumvent this problem
>>>>> Thanks
>>>>> Mike
>>>>>
>>>>>
>>>>> On Mon, May 27, 2013 at 10:46 PM, Michael Stewart
>>>>> <[email protected]> wrote:
>>>>>> HI ,
>>>>>> First of all, thank you very much Steve and Nick.It works like a charm
>>>>>> and I can automate stata to replicate many tasks
>>>>>> I work on Windows .Using ``x'' works.I havent tested `x'.But it
>>>>>> Works.Thats great .
>>>>>> Thank you Steve and Nick
>>>>>> Thank you ,
>>>>>> Yours Sincerely,
>>>>>> Mike.
>>>>>>
>>>>>>
>>>>>> On Mon, May 27, 2013 at 9:46 PM, Steve Samuels <[email protected]> wrote:
>>>>>>>
>>>>>>> Well, I was concerned about that, but the code worked for me in
>>>>>>> Mac OS X, Here was my test do file:
>>>>>>>
>>>>>>>
>>>>>>> **********************************
>>>>>>> local ds1 "~Downloads/testsim01"
>>>>>>> local ds2 "~Downloads/dtest01"
>>>>>>>
>>>>>>> local all ds1 ds2
>>>>>>> di `"`all'"'
>>>>>>> foreach x in `all' {
>>>>>>> use `x', clear
>>>>>>> gen nv =1
>>>>>>> save `x'_new, replace
>>>>>>> }
>>>>>>> ********************************
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>>> On May 27, 2013, at 8:50 PM, Nick Cox wrote:
>>>>>>>
>>>>>>> This won't work, as when Stata hits the loop
>>>>>>>
>>>>>>> foreach x in `all'
>>>>>>>
>>>>>>> it will then see
>>>>>>>
>>>>>>> foreach x in ds1 ds2
>>>>>>>
>>>>>>> and will fail at
>>>>>>>
>>>>>>> use ds1, clear
>>>>>>>
>>>>>>> There are two levels of macro referencing in Steve's code, but only
>>>>>>> one level going backwards.
>>>>>>>
>>>>>>> use ``x'', clear
>>>>>>>
>>>>>>> is one way to fix it.
>>>>>>>
>>>>>>> Nick
>>>>>>> [email protected]
>>>>>>>
>>>>>>>
>>>>>>> On 28 May 2013 01:23, Steve Samuels <[email protected]> wrote:
>>>>>>>>
>>>>>>>> Michael Steward ([email protected]):
>>>>>>>>
>>>>>>>> You can loop, as Nick suggests. Put the data management tasks in
>>>>>>>> external do files, e.g. tasks1.do, tasks2.do, tasks3.do. Then something
>>>>>>>> like the following should work (not tested in Windows). See the help for
>>>>>>>> -include-. The big problem I anticipate is the mounds of output that
>>>>>>>> this could generate.
>>>>>>>>
>>>>>>>>
>>>>>>>> *********************************
>>>>>>>> local ds1 "c/data/dataset-1"
>>>>>>>> local ds2 "c/data/newdata/dataset-2" // etc.
>>>>>>>>
>>>>>>>> local all ds1 ds2
>>>>>>>>
>>>>>>>> foreach x in `all' {
>>>>>>>> use `x', clear
>>>>>>>>
>>>>>>>> include tasks1
>>>>>>>> include tasks2
>>>>>>>> include tasks3
>>>>>>>>
>>>>>>>> save `x'_new, replace
>>>>>>>> }
>>>>>>>> *********************************
>>>>>>>>
>>>>>>>> Steve
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 27, 2013, at 3:19 PM, Michael Stewart wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>> I am trying to find a way to work multiple files successively.
>>>>>>>> I have multiple large files(really large) placed in different directories
>>>>>>>> I am trying to find a way to load each file into stata, complete bunch
>>>>>>>> of same datamagement tasks,create a new file from it and places them
>>>>>>>> in a common folder
>>>>>>>>
>>>>>>>> I could use a dofile and do something like
>>>>>>>>
>>>>>>>> clear
>>>>>>>> use c\data\dataset-1
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> save c\data\dataset-1_new
>>>>>>>> clear
>>>>>>>> use c\data\newdata\dataset-2
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> save c\data\newdata\dataset-2_new
>>>>>>>> clear
>>>>>>>>
>>>>>>>> But is there an elegant and simpler way to do it given that I have
>>>>>>>> nearly 15 files to do data management tasks.
>>>>>>>>
>>>>>>>> Can we use foreach loop / forvalues loop or anything like that .Please suggest
>>>> *
>>>> * For searches and help try:
>>>> * http://www.stata.com/help.cgi?search
>>>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> * http://www.ats.ucla.edu/stat/stata/
>>>
>>>
>>>
>>> --
>>> Thank you ,
>>> Yours Sincerely,
>>> Mike.
>>> *
>>> * For searches and help try:
>>> * http://www.stata.com/help.cgi?search
>>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>>> * http://www.ats.ucla.edu/stat/stata/
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>> * http://www.ats.ucla.edu/stat/stata/
>
>
>
> --
> Thank you ,
> Yours Sincerely,
> Mike.
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/faqs/resources/statalist-faq/
> * http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/