RE: st: dynamic line execution in mata

[Andrew Maurer <Andrew.Maurer@qrm.com>]

Thanks for the response, Nick. I looked into pointers and have been able to make use of them. I'll give the background of the problem. I would be very interested to hear if anyone has thoughts on the efficiency of the code I have so far (see bottom of post).

I am writing a Stata program, saveif, that will save a subset of observations of a dataset to a file. One method to accomplish this would be to do something like:
preserve
keep if...
save...
restore

However, for large datasets (eg 20gb) and few observations to be saved (eg - a few mb of outliers), I expect that the preserve/restore method is grossly inefficient, since it involves writing the entire dataset from memory to hard-disk, then reading it back from hard-disk to memory.

An alternative method to accomplish the task would be to somewhat manually "file write" the individual observations to a file, without having to clear and load back the dataset from memory. I have a nearly complete example here, where there is one part that has been hard-coded to the specific example of gnp96.dta. The code is still somewhat rough.

Thank you,
Andrew Maurer 



* Want to write a program that will save a set of observations into a dataset
mata: mata clear
clear all

cap program drop saveif
program define saveif
	syntax varlist [if] [in] using/, [replace]
	putmata `varlist' `if' `in', view
	
	* put a row vector called varnames to mata
	forval i = 1/`: word count `varlist'' {
		if `i' == 1 {
			mata: varnames = "`: word `i' of `varlist''"
			mata: vartypes = "`: type `: word `i' of `varlist'''"
			mata: varpointers = &`: word `i' of `varlist'' // pointers
		}
		else {
			mata: varnames = varnames,"`: word `i' of `varlist''"
			mata: vartypes = vartypes,"`: type `: word `i' of `varlist'''"
			mata: varpointers = varpointers,&`: word `i' of `varlist'' // pointers
		}
	}
	* save vector of varnames to file
	cap confirm new file "`using'"
	if _rc != 0 {
		di as error "`using' exists. replacing"
		rm "`using'"
	}
	mata: fh = fopen("`using'", "w")
	mata: fputmatrix(fh, varnames)
	mata: fputmatrix(fh, vartypes)
	mata: fputmatrix(fh, varpointers)
		
	* write observations of each variable to file
	forval i = 1/`: word count `varlist'' {
		mata: fputmatrix(fh, `: word `i' of `varlist'')
	}
	
	mata: fclose(fh)
end	


capture mata mata drop recover_from_saveif()
mata:
void recover_from_saveif(string fileloc)
{

	fh = fopen(fileloc, "r")
	varnames = fgetmatrix(fh) 
	vartypes = fgetmatrix(fh) 
	varpointers = fgetmatrix(fh) 
	// ----- hard coded part!! try to get this into loop
	date = fgetmatrix(fh) 
	gnp96 = fgetmatrix(fh) 
	// -------------------------------------------------
	fclose(fh)
	varcount = cols(varnames)

	// ------- this loop not working yet. need to figure out syntax
	// foreach var of varnames, read var from file to mata
	for (i=1; i<=varcount;i++) {
	  // varnames[1,i] = fgetmatrix(fh) 
	}
	// -------------------------------------------------
	
	// foreach var of varnames, load var into stata with correct variable type
	for (i=1; i<=varcount;i++) {
		thisvarname = varnames[1,i] // eg contains "date"
		thisvartype = vartypes[1,i] // eg contains "int"
		thisvar = varpointers[1,i] // eg pointer to date vector
		if (i == 1) st_addobs(rows(*thisvar))
		st_store(., st_addvar(thisvartype,thisvarname),*thisvar) 
	}

}
end

cap program drop recover_from_saveif
program define recover_from_saveif
	syntax using/, [replace]
	
	mata: recover_from_saveif("`using'")
	
end


sysuse gnp96.dta, clear

saveif * in 1/5 using test5.txt

clear
recover_from_saveif using test5.txt














-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Monday, February 10, 2014 11:59 AM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: dynamic line execution in mata

You are presuming that such a thing exists.

In essence, Mata has no direct equivalent of macro substitution.

Sometimes, the way to solve (similar) problems is by direct manipulation of strings. That is the theme of

SJ-11-2 pr0052  . . . . Stata tip 100: Mata and the case of the missing macros
        . . . . . . . . . . . . . . . . . . . . . . . . W. Gould and N. J. Cox
        Q2/11   SJ 11(2):323--324                                (no commands)
        tip showing how to do the equivalent of Stata's macro
        substitution in Mata

Sometimes, using pointers is the answer.

In this case, I'd guess that you want the Mata equivalent of some Stata operation and that there's a Mata way of doing it, but I would rather hear whether that is so than try to guess what the underlying problem is.

Nick
njcoxstata@gmail.com


On 10 February 2014 17:38, Andrew Maurer <Andrew.Maurer@qrm.com> wrote:
> Hi Statalist,
>
> I am trying to find Mata's equivalent of Stata's macro expansion functionality. In the below example, I first define an object thisvar as the string "date" and I define the object date as the column vector 1 \ 2 \ 3 \ 4 \ 5. How can I return the contents of the "date" object by only referencing "thisvar"?
>
> In the line, rows( thisvar ), thisvar is simply the 1x1 matrix containing the string "date", so rows( thisvar ) returns: 1. What I am looking for is something like rows( `=thisvar' ), so as to return 5 rather than 1.
>
> ********* begin example *********
>
> mata
>
> i = 1
> date = 1 \ 2 \ 3 \ 4 \ 5
> varnames = "date", "price"
> thisvar = varnames[1,i]
> rows( thisvar ) // output: 1
> rows( date ) // output: 5
>
> end
>
> ********* end example ***********
>
> Thank you,
> Andrew Maurer
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/