Re: st: How to reference results from a big dataset within a program

[Joerg Luedicke <joerg.luedicke@gmail.com>]

Thanks for the clarification. I was just wondering if it would be
possible to work out an analytical solution for what you otherwise
obtain from your "long list of results", so that you could just
calculate whatever you need for your next step and ditch the reference
to the set of "results" altogether. But maybe it is not feasible for
your problem, I don't know.

Joerg

On Wed, Aug 28, 2013 at 12:46 PM, Chen,Minxing <MChen3@mdanderson.org> wrote:
> Hi Joerg, the "pre-simulated dataset" I mentioned early is a really long list of results I want from simulated joint binary probabilities. The reason I didn't want to do this simulation within my program is it will take significantly long time (e.g. many hours). Instead of doing the simulation within my program and get the result I need for next step, I thought referencing the result from the data I included with the program will be much easier. Minxing
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Joerg Luedicke
> Sent: Tuesday, August 27, 2013 4:46 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: How to reference results from a big dataset within a program
>
> What do you mean with "pre-simulated dataset"? Is it _one_ fabricated dataset, or results from 400k replications of a Monte Carlo simulation?
>
> Either way, it seems a bit puzzling why you would even need to include a fixed set of data in your program? What information exactly are you pulling from this dataset? Depending on the specific purpose of including these data in your program, maybe you could derive some analytic solution?
>
> Joerg
>
>
> On Tue, Aug 27, 2013 at 5:25 PM, Chen,Minxing <MChen3@mdanderson.org> wrote:
>> Dear all,
>>
>> I submitted a manuscript and program to Stat Journal several months ago, and now got feedback from reviewer. I need a big help from your all regarding how to reference a big dataset within a program.
>>
>> Basically, in the program I submitted, I had to reference results from a big pre-simulated dataset (four variables, but around 400,000 observations). In my previous submission, I simply submitted the pre-simulated dataset with my program, and within the program I called up that simulated dataset by using code such as " use c:\ado\personal\simudata". I was hoping when people download the program from SSC, the pre-simulated dataset will be also downloaded to the directory "c:\ado\personal\".
>>
>> Now my reviewer indicated that I can't expect users to do that, I can't even tell the user to place the file there because such a directory may not be creatable for the user (e.g. they might not have a C: drive). The reviewer suggested me to find some other way to get the information in my pre-simulated dataset, such as incorporating the data into the program.
>>
>> I tried to copy of the simulated data within my program by using syntax such as "input x y z k", however, since there are so many observations (a little more than 400,000), and there are system limit for the maximum lines of syntax within a program (around 3500), I was not able to do this way. The reviewer also mentioned that I may use "Mata library" function, but I am pretty new to Stata Mata. Is there anyone that may be able to help regarding this issue?
>>
>> Thank you so much!
>> Minxing Chen
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/