Re: st: Need help with working with multiple files

[Michael Stewart <michaelstewartresearch@gmail.com>]

Dear Statalist user ,
I am having problem with saving files after damamanagement tasks using
a locals in foreach loop and facing difficulty.(As I try to work with
three datasets, the output is only one dataset).I have tried many
options, sought help of statlist but able to correct the problem
partially but there is a bug somewhere.  I am here with posting the
code and stata output with "traceon".May be any one could please help
me.

MY CODE / DO FILE IS AS FOLLOWS
local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
local all ds1 ds2 ds3
foreach x in `all'{
use ``x'',clear
di "``x''"
include "D:\_Dummy\multiple\4\mpg.do"
di "``x''"
include "D:\_Dummy\multiple\5\sum.do"
save ``x''_new.dta,replace
}

mpg DO FILE IS AS  FOLLOWS ( NO FOREACH LOOP IN THIS DOFILE)
gen mean=sum(mpg)

sum DOFILE IS AS FOLLOWS  (FOREACH LOOP PRESENT  IN THIS DOFILE)
foreach x of varlist price mpg rep78 trunk{
gen x_`x'=`x'==3
}


PROBLEM: instead of generating three new  datasets , it is generating
one dataset _new.

MY STATA OUTPUT IS AS FOLLOWS(WITH TRACE ON AND DEBUGGNG STEP di
``x''); LOOKS LIKE AFTER EXECUTUNG THE SECOND DOFILE (WITH FOREACH
LOOP) di ``x''  shows nothing!!!!!

****************************************************************************************************************************
 local ds1 "D:\_Dummy\multiple\1\auto_1.dta"

. local ds2 "D:\_Dummy\multiple\2\auto_2.dta"

. local ds3 "D:\_Dummy\multiple\3\auto_3.dta"

. local all ds1 ds2 ds3

. foreach x in `all'{
  2. use ``x'',clear
  3. di "``x''"
  4. include "D:\_Dummy\multiple\4\mpg.do"
  5. di "``x''"
  6. include "D:\_Dummy\multiple\5\sum.do"
  7. save ``x''_new.dta,replace
  8. }
- foreach x in `all'{
= foreach x in ds1 ds2 ds3{
- use ``x'',clear
= use D:\_Dummy\multiple\1\auto_1.dta,clear
(1978 Automobile Data)
- di "``x''"
= di "D:\_Dummy\multiple\1\auto_1.dta"
D:\_Dummy\multiple\1\auto_1.dta
- include "D:\_Dummy\multiple\4\mpg.do"

. gen mean=sum(mpg)

.
. - di "``x''"
= di "D:\_Dummy\multiple\1\auto_1.dta"
D:\_Dummy\multiple\1\auto_1.dta
- include "D:\_Dummy\multiple\5\sum.do"

. gen xx= sum(weight)

. foreach x of varlist price mpg rep78 trunk{
  2. gen x_`x'=`x'==3
  3. }
  - foreach x of varlist price mpg rep78 trunk{
  - gen x_`x'=`x'==3
  = gen x_price=price==3
  - }
  - gen x_`x'=`x'==3
  = gen x_mpg=mpg==3
  - }
  - gen x_`x'=`x'==3
  = gen x_rep78=rep78==3
  - }
  - gen x_`x'=`x'==3
  = gen x_trunk=trunk==3
  - }

.
. - save ``x''_new.dta,replace
= save _new.dta,replace
file _new.dta saved
- }
- use ``x'',clear
= use D:\_Dummy\multiple\2\auto_2.dta,clear
(1978 Automobile Data)
- di "``x''"
= di "D:\_Dummy\multiple\2\auto_2.dta"
D:\_Dummy\multiple\2\auto_2.dta
- include "D:\_Dummy\multiple\4\mpg.do"

. gen mean=sum(mpg)

.
. - di "``x''"
= di "D:\_Dummy\multiple\2\auto_2.dta"
D:\_Dummy\multiple\2\auto_2.dta
- include "D:\_Dummy\multiple\5\sum.do"

. gen xx= sum(weight)

. foreach x of varlist price mpg rep78 trunk{
  2. gen x_`x'=`x'==3
  3. }
  - foreach x of varlist price mpg rep78 trunk{
  - gen x_`x'=`x'==3
  = gen x_price=price==3
  - }
  - gen x_`x'=`x'==3
  = gen x_mpg=mpg==3
  - }
  - gen x_`x'=`x'==3
  = gen x_rep78=rep78==3
  - }
  - gen x_`x'=`x'==3
  = gen x_trunk=trunk==3
  - }

.
. - save ``x''_new.dta,replace
= save _new.dta,replace
file _new.dta saved
- }
- use ``x'',clear
= use D:\_Dummy\multiple\3\auto_3.dta,clear
(1978 Automobile Data)
- di "``x''"
= di "D:\_Dummy\multiple\3\auto_3.dta"
D:\_Dummy\multiple\3\auto_3.dta
- include "D:\_Dummy\multiple\4\mpg.do"

. gen mean=sum(mpg)

.
. - di "``x''"
= di "D:\_Dummy\multiple\3\auto_3.dta"
D:\_Dummy\multiple\3\auto_3.dta
- include "D:\_Dummy\multiple\5\sum.do"

. gen xx= sum(weight)

. foreach x of varlist price mpg rep78 trunk{
  2. gen x_`x'=`x'==3
  3. }
  - foreach x of varlist price mpg rep78 trunk{
  - gen x_`x'=`x'==3
  = gen x_price=price==3
  - }
  - gen x_`x'=`x'==3
  = gen x_mpg=mpg==3
  - }
  - gen x_`x'=`x'==3
  = gen x_rep78=rep78==3
  - }
  - gen x_`x'=`x'==3
  = gen x_trunk=trunk==3
  - }

.
. - save ``x''_new.dta,replace
= save _new.dta,replace
file _new.dta saved
- }

.
end of do-file
****************************************************************************************************************************


So as you have noticed a, after executing dofile  "sum", the naleof
the file is being removed and di "" shows nothing.

Can anyone of the users help fix the bug.

Thank you
Mike




On Tue, May 28, 2013 at 7:37 AM, Nick Cox <njcoxstata@gmail.com> wrote:
> It is best to think that you are addressing Statalist rather than
> think that one person is necessarily going to answer. Assume that
> people at the other end are just as capricious as you might be and get
> bored, need to meet partners, attend to the day job, or write their
> novel instead. On the whole the list is quite good at picking up
> answerable questions but we don't have a buddy system.
>
> From your example it seems likeliest  that one of the extra  loops you
> are including is somehow zapping the local macro x. I don't have
> another guess. Inner loops are not in principle fatal to outer loops
> -- otherwise programming would hardly be possible -- but it seems that
> your code is stomping on itself.
>
> Your code is longer and more complicated than you care to show and you
> can't win easily here. Post too much code or too little; either can be
> wrong in terms of not getting a useful reply.
>
> But I would check that you are not using the same macro name for
> different loops.
>
> Also, you can add debugging steps
>
> di "`mymacro'"
>
> to get Stata to show what it thinks -mymacro- is (or is not).
>
> Example where nested loops are fine.
>
> . foreach x in a b c {
>   2.     di "first time   `x'"
>   3.         foreach y in 1 2 3 {
>   4.                 di "`y'"
>   5.         }
>   6.         di "second time  `x'"
>   7. }
> first time   a
> 1
> 2
> 3
> second time  a
> first time   b
> 1
> 2
> 3
> second time  b
> first time   c
> 1
> 2
> 3
> second time  c
> Nick
> njcoxstata@gmail.com
>
>
> On 28 May 2013 11:24, Michael Stewart <michaelstewartresearch@gmail.com> wrote:
>> HI Nick
>> Sorry to keep bothering you as I wanted to clarify the last few
>> sentences of your reply "save the file within
>> the loop _or_ otherwise use the name you want directly."
>>
>> Here in my situation, the code would be
>>
>> clear
>> local ds1 c\data\top/dataset
>> local ds2  c\data\date/news/dataset
>> ...
>> ...
>> ..
>> local ds13  c\data\data/datedataset-
>> local all ds1 ....ds13
>> foreach x in `all'{
>> use ``x'',clear
>> datamagement tasks   / contains foreaach loop
>> datamagement tasks  / contains foreaach loop
>> datamagement tasks  / contains foreaach loop
>>
>> save ?????
>> }
>>
>> Here while saving the dataset , when I use  save ``x''_new,replace-->
>> this saves the dataset as _new and we get one dataset instead of the
>> 13 dataset that we started with
>>
>> According to your recommendation "otherwise use the name you want
>> directly", how can I fit 13 names here in the loop .
>> Am I missing some  important concept here?? Please help me with  the
>> problem and understand the concept.
>>
>> Thanks
>> MIke
>>
>>
>> On Tue, May 28, 2013 at 3:40 AM, Nick Cox <njcoxstata@gmail.com> wrote:
>>> If you refer to a macro that doesn't exist Stata doesn't regard that
>>> as an error: it just substitutes nothing, or equivalently ignores the
>>> reference.
>>>
>>> Nested macro references don't affect this principle and indeed must be
>>> consistent with it. So if  local macro x does not exist as a string to
>>> be used neither do any of
>>>
>>> `x'
>>> ``x''
>>> ```x''''
>>>
>>> and so forth. You could say that they define empty strings if you
>>> like, but the effect is the same: a non-existent string and an empty
>>> string look identical.
>>>
>>> From what you say that is precisely what you did. It was the -foreach-
>>> loop that defined x as what the loop used -- and (an extra principle
>>> here) that macro is not defined outside the loop, and indeed is doubly
>>> local, to the loop as well as in the usual sense.
>>>
>>> The user manual [U] contains a basic introduction to macros in
>>>
>>> [U]     Chapter 18  . . . . . . . . . . . . . . . . . . . .  Programming Stata
>>>         (help program, macro, quotes, syntax, examplehelpfile)
>>>
>>> From your earlier posts it seems that you need to save the file within
>>> the loop _or_ otherwise use the name you want directly.
>>>
>>> Nick
>>> njcoxstata@gmail.com
>>>
>>> On 28 May 2013 06:00, Michael Stewart <michaelstewartresearch@gmail.com> wrote:
>>>> Dear Steve and Nick ,
>>>> I am having problem with saving the files
>>>> I am using ``x'' and the dofile runs through multiple dofiles like a charm
>>>> However, after completetion of the datamanagement tasks, the  each
>>>> file is being stored as "_new" in the stata working directory.
>>>> And the interesting part is that  this happens only if the individual
>>>> tasks have foreach loop in it. If the individual tasks dont have
>>>> foreach loop in it, then the files are saved in respective folders
>>>> i.e., save ``x''_new, replace --> works
>>>> I am not sure is there is a way to circumvent this problem
>>>> Thanks
>>>> Mike
>>>>
>>>>
>>>> On Mon, May 27, 2013 at 10:46 PM, Michael Stewart
>>>> <michaelstewartresearch@gmail.com> wrote:
>>>>> HI ,
>>>>> First of all, thank you very much Steve and Nick.It works like a charm
>>>>> and I can automate stata to replicate many tasks
>>>>> I work on Windows .Using ``x'' works.I havent tested `x'.But it
>>>>> Works.Thats great .
>>>>> Thank you Steve and Nick
>>>>> Thank you ,
>>>>> Yours Sincerely,
>>>>> Mike.
>>>>>
>>>>>
>>>>> On Mon, May 27, 2013 at 9:46 PM, Steve Samuels <sjsamuels@gmail.com> wrote:
>>>>>>
>>>>>> Well, I was concerned about that, but the code worked for me in
>>>>>> Mac OS X, Here was my test do file:
>>>>>>
>>>>>>
>>>>>> **********************************
>>>>>> local ds1 "~Downloads/testsim01"
>>>>>> local ds2 "~Downloads/dtest01"
>>>>>>
>>>>>> local all ds1 ds2
>>>>>> di `"`all'"'
>>>>>> foreach x in `all' {
>>>>>> use `x', clear
>>>>>> gen nv =1
>>>>>> save `x'_new, replace
>>>>>> }
>>>>>> ********************************
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>> On May 27, 2013, at 8:50 PM, Nick Cox wrote:
>>>>>>
>>>>>> This won't work, as when Stata hits the loop
>>>>>>
>>>>>> foreach x in `all'
>>>>>>
>>>>>> it will then see
>>>>>>
>>>>>> foreach x in ds1 ds2
>>>>>>
>>>>>> and will fail at
>>>>>>
>>>>>> use ds1, clear
>>>>>>
>>>>>> There are two levels of macro referencing in Steve's code, but only
>>>>>> one level going backwards.
>>>>>>
>>>>>> use ``x'', clear
>>>>>>
>>>>>> is one way to fix it.
>>>>>>
>>>>>> Nick
>>>>>> njcoxstata@gmail.com
>>>>>>
>>>>>>
>>>>>> On 28 May 2013 01:23, Steve Samuels <sjsamuels@gmail.com> wrote:
>>>>>>>
>>>>>>> Michael Steward (michaelstewartresearch@gmail.com):
>>>>>>>
>>>>>>> You can loop, as Nick suggests. Put the data management tasks in
>>>>>>> external do files, e.g. tasks1.do, tasks2.do, tasks3.do. Then something
>>>>>>> like the following should work (not tested in Windows). See the help for
>>>>>>> -include-. The big problem I anticipate is the mounds of output that
>>>>>>> this could generate.
>>>>>>>
>>>>>>>
>>>>>>> *********************************
>>>>>>> local ds1 "c/data/dataset-1"
>>>>>>> local ds2 "c/data/newdata/dataset-2" // etc.
>>>>>>>
>>>>>>> local all ds1 ds2
>>>>>>>
>>>>>>> foreach x in `all' {
>>>>>>>   use `x', clear
>>>>>>>
>>>>>>>   include tasks1
>>>>>>>   include tasks2
>>>>>>>   include tasks3
>>>>>>>
>>>>>>>   save `x'_new, replace
>>>>>>> }
>>>>>>> *********************************
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>>>
>>>>>>> On May 27, 2013, at 3:19 PM, Michael Stewart wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>> I am trying to find a way to work multiple files successively.
>>>>>>> I have multiple large files(really large) placed in different directories
>>>>>>> I am trying to find a way to load each file into stata, complete bunch
>>>>>>> of same datamagement tasks,create  a new file from it and places them
>>>>>>> in a common folder
>>>>>>>
>>>>>>> I could use a dofile and do something like
>>>>>>>
>>>>>>> clear
>>>>>>> use c\data\dataset-1
>>>>>>> datamagement tasks
>>>>>>> datamagement tasks
>>>>>>> datamagement tasks
>>>>>>> save c\data\dataset-1_new
>>>>>>> clear
>>>>>>> use c\data\newdata\dataset-2
>>>>>>> datamagement tasks
>>>>>>> datamagement tasks
>>>>>>> datamagement tasks
>>>>>>> save c\data\newdata\dataset-2_new
>>>>>>> clear
>>>>>>>
>>>>>>> But is there an elegant and simpler way to do it given that I have
>>>>>>> nearly 15 files to do data management tasks.
>>>>>>>
>>>>>>> Can we use foreach loop / forvalues loop or anything like that .Please suggest
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>>
>>
>>
>> --
>> Thank you ,
>> Yours Sincerely,
>> Mike.
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/



--
Thank you ,
Yours Sincerely,
Mike.
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/