Re: st: -project- command and "inverse" table of file dependencies

[Roberto Ferrer <refp16@gmail.com>]

Dear Robert,

Many thanks for your reply.

1. I completely missed -list(build) which is exactly what I was after.
The alphabetical listing seems a good idea, as well as an "only
do-file" or "only created files" (with corresponding do-files) listing
that could maybe come in the form of options.

2. About unnecessary duplicate entries I just meant it was just the
consequence of using -list(concordance)- (along with stub-naming and
reading backwards) for what I wanted. For example, below I can see
that the .smcl and .png files are associated with -separate.do- but
the latter appears as many times as associated files there are (here
two). Never mind that.
-------------------------------------------------------------------------------------
|separate.smcl             7,923   714867776                  |
| -->  separate.do            610  1479884638   creates  |
|-----------------------------------------------------------------------------------|
| separate_example1.png    61,465  3032149987       |
| -->  separate.do            610  1479884638   creates  |
|-------------------------------------------------------------------------------------

3. The feature I was asking to be included was precisely -list(build)-.

Very nice package.

Regards,
Roberto


On Sat, Jul 20, 2013 at 4:38 PM, Robert Picard <picard@netbox.com> wrote:
> Glad to hear that you find -project- useful and thanks for the
> comments/questions. See below for my specific comments.
>
>
> On Sat, Jul 20, 2013 at 10:05 AM, Roberto Ferrer <refp16@gmail.com> wrote:
>> I'm very enthusiastic about the user-written package -project- by
>> Robert Picard and installed using:
>>
>>  net from http://robertpicard.com/stata
>>
>> Reproducibility of work and simple organization are my main concerns.
>> I believe this package aids the process wonderfully.
>>
>> I've been playing around with the example provided with the package
>> (ex12 project) and  I ask if it is capable of producing an "inverse"
>> table of file dependencies. The command:
>>
>> project ex12, list(concordance)
>>
>> produces file_created/used --> do-file:
>>
>> |separate.smcl             7,923   714867776            20jul2013
>> 10:40:48  data-management                      |
>> | -->  separate.do            610  1479884638   creates  20jul2013
>> 10:40:46  data-management                     |
>> |-----------------------------------------------------------------------------------------------------------------|
>> | separate_example1.png    61,465  3032149987            20jul2013
>> 10:40:48  data-management                     |
>> | -->  separate.do            610  1479884638   creates  20jul2013
>> 10:40:46  data-management                     |
>> |----------------------------------------------------------------------------------------------------------------
>>
>> I'd like to see an "inverted" table that shows do-file --> file_created/used:
>>
>> |separate.do                 7,923   714867776            20jul2013
>> 10:40:48  data-management                    |
>> | -->  separate.smcl            610  1479884638   creates  20jul2013
>> 10:40:46  data-management                   |
>> | -->  separate_example1.png    610  1479884638   creates  20jul2013
>> 10:40:46  data-management                   |
>> | -->  result_data.dta          xxxxxxxxxxxxxxx   creates  20jul2013
>> 10:40:46  YYYYYYYYYYYYYYYYY                 |
>> | -->  proc _data.dta          xxxxxxxxxxxxxxx   uses      20jul2013
>> 10:40:46  YYYYYYYYYYYYYYYYY                 |
>> | -->  raw_data.dta            xxxxxxxxxxxxxxx   original  20jul2013
>> 10:40:46  YYYYYYYYYYYYYYYYY                 |
>> | -->  some_name.do          xxxxxxxxxxxxxxx     uses      20jul2013
>> 10:40:46  YYYYYYYYYYYYYYYYY                 |
>> |-----------------------------------------------------------------------------------------------------------------|
>>
>
> With respect to your request for an "inverse" of the
> -list(concordance)- option, that's pretty much what -list(build)-
> does, i.e. list all dependencies per do-file, as they appear in the
> build. I can see that this is not organized in an alphabetical list of
> do-files so I'll note your suggestion and most likely add the feature
> in a future release.
>
>> In its current state I think the naming convention using stubs:
>>
>> local dofile "`r(dofile)'"        // do-file's stub name
>> graph export "`dofile'_example1.png", width(1200) replace
>>
>> may be of aid, but it forces the user to name every single file with
>> the do-file stub name, which may prove inconvenient at some point. It
>> also shows unnecessary duplicate entries in the table and forces
>> "backward visualization".
>
> I find using stubs based on do-file names useful because I change my
> mind constantly about do-file names as each project evolves. If
> "table1.do" becomes "table2.do", then I don't need to adjust any of
> the files created by the do-file. I don't understand your comment
> about unnecessary duplicate entries. Note that you are not forced to
> use stub names, just use a naming convention that suits your style,
> e.g. -graph export "sep_ex1.png", width(1200) replace-
>
>>
>> Is this possible right now?
>>
>> To the author: If not, what are the odds of including this feature in
>> the near future?
>
> What feature is that?
>
>
>>
>> A related point is the dependence of do-files on other do-files. I
>> think it would be nice if -project, do(file_name)- recorded the
>> dependency between file_name within the command and the do-file that
>> contains the command (at least as an option). In the inverted table
>> the last dependency corresponds to such a case. For example, the ex12
>> example project does not show the dependency of d_examples.do on all
>> the do-files it runs. One of the things I'd like to see when given
>> some unknown zipped project is precisely this dependency. Is there any
>> reason for this not being implemented?
>
> Again, this goes back to -list(build)- which shows exactly how
> do-files are linked to each others, using tabbed columns to show the
> level of nested do-files. If there is a change in the code of
> "separate.do", then it will run again in the next build. If you follow
> the -list(build)- output backwards from "separate.do", you see that it
> is called by "d_examples.do" wich is itself called by "ex12.do". I
> suppose that a shorter listing with just do-files might give a better
> visual. I'll put this on my list of things to think about.
>
>>
>> Finally, there's a small typo in the ex12 example project file
>> ex12/base/r_examples.do. the author might want to correct. Line 4
>> reads:
>>
>> Run examples from the Data-Management reference manual
>>
>> and should read:
>>
>> Run examples from the Base reference manual.
>
> Thanks and will be corrected in the next release. It shows that in
> real life, new do-files often start their lives as copies of similar
> past work.
>
>>
>> Thanks.
>>
>> Roberto
>> Stata 12.1 (64-bit)
>> Linux Mint Debian Edition (LMDE)
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/