Re: st: -project- command and "inverse" table of file dependencies

[Robert Picard <picard@netbox.com>]

Glad to hear that you find -project- useful and thanks for the
comments/questions. See below for my specific comments.


On Sat, Jul 20, 2013 at 10:05 AM, Roberto Ferrer <refp16@gmail.com> wrote:
> I'm very enthusiastic about the user-written package -project- by
> Robert Picard and installed using:
>
>  net from http://robertpicard.com/stata
>
> Reproducibility of work and simple organization are my main concerns.
> I believe this package aids the process wonderfully.
>
> I've been playing around with the example provided with the package
> (ex12 project) and  I ask if it is capable of producing an "inverse"
> table of file dependencies. The command:
>
> project ex12, list(concordance)
>
> produces file_created/used --> do-file:
>
> |separate.smcl             7,923   714867776            20jul2013
> 10:40:48  data-management                      |
> | -->  separate.do            610  1479884638   creates  20jul2013
> 10:40:46  data-management                     |
> |-----------------------------------------------------------------------------------------------------------------|
> | separate_example1.png    61,465  3032149987            20jul2013
> 10:40:48  data-management                     |
> | -->  separate.do            610  1479884638   creates  20jul2013
> 10:40:46  data-management                     |
> |----------------------------------------------------------------------------------------------------------------
>
> I'd like to see an "inverted" table that shows do-file --> file_created/used:
>
> |separate.do                 7,923   714867776            20jul2013
> 10:40:48  data-management                    |
> | -->  separate.smcl            610  1479884638   creates  20jul2013
> 10:40:46  data-management                   |
> | -->  separate_example1.png    610  1479884638   creates  20jul2013
> 10:40:46  data-management                   |
> | -->  result_data.dta          xxxxxxxxxxxxxxx   creates  20jul2013
> 10:40:46  YYYYYYYYYYYYYYYYY                 |
> | -->  proc _data.dta          xxxxxxxxxxxxxxx   uses      20jul2013
> 10:40:46  YYYYYYYYYYYYYYYYY                 |
> | -->  raw_data.dta            xxxxxxxxxxxxxxx   original  20jul2013
> 10:40:46  YYYYYYYYYYYYYYYYY                 |
> | -->  some_name.do          xxxxxxxxxxxxxxx     uses      20jul2013
> 10:40:46  YYYYYYYYYYYYYYYYY                 |
> |-----------------------------------------------------------------------------------------------------------------|
>

With respect to your request for an "inverse" of the
-list(concordance)- option, that's pretty much what -list(build)-
does, i.e. list all dependencies per do-file, as they appear in the
build. I can see that this is not organized in an alphabetical list of
do-files so I'll note your suggestion and most likely add the feature
in a future release.

> In its current state I think the naming convention using stubs:
>
> local dofile "`r(dofile)'"        // do-file's stub name
> graph export "`dofile'_example1.png", width(1200) replace
>
> may be of aid, but it forces the user to name every single file with
> the do-file stub name, which may prove inconvenient at some point. It
> also shows unnecessary duplicate entries in the table and forces
> "backward visualization".

I find using stubs based on do-file names useful because I change my
mind constantly about do-file names as each project evolves. If
"table1.do" becomes "table2.do", then I don't need to adjust any of
the files created by the do-file. I don't understand your comment
about unnecessary duplicate entries. Note that you are not forced to
use stub names, just use a naming convention that suits your style,
e.g. -graph export "sep_ex1.png", width(1200) replace-

>
> Is this possible right now?
>
> To the author: If not, what are the odds of including this feature in
> the near future?

What feature is that?


>
> A related point is the dependence of do-files on other do-files. I
> think it would be nice if -project, do(file_name)- recorded the
> dependency between file_name within the command and the do-file that
> contains the command (at least as an option). In the inverted table
> the last dependency corresponds to such a case. For example, the ex12
> example project does not show the dependency of d_examples.do on all
> the do-files it runs. One of the things I'd like to see when given
> some unknown zipped project is precisely this dependency. Is there any
> reason for this not being implemented?

Again, this goes back to -list(build)- which shows exactly how
do-files are linked to each others, using tabbed columns to show the
level of nested do-files. If there is a change in the code of
"separate.do", then it will run again in the next build. If you follow
the -list(build)- output backwards from "separate.do", you see that it
is called by "d_examples.do" wich is itself called by "ex12.do". I
suppose that a shorter listing with just do-files might give a better
visual. I'll put this on my list of things to think about.

>
> Finally, there's a small typo in the ex12 example project file
> ex12/base/r_examples.do. the author might want to correct. Line 4
> reads:
>
> Run examples from the Data-Management reference manual
>
> and should read:
>
> Run examples from the Base reference manual.

Thanks and will be corrected in the next release. It shows that in
real life, new do-files often start their lives as copies of similar
past work.

>
> Thanks.
>
> Roberto
> Stata 12.1 (64-bit)
> Linux Mint Debian Edition (LMDE)
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/