Re: st: using the first n observations in a dataset w/o evaluating the whole thing?

[David Kantor <kantor.d@att.net>]

At 07:31 PM 4/3/2008, Mark Rodini wrote:
> Greetings.
>
> Suppose I have a large Stata dataset (e.g. 3,000,000 observations) and I
> only with to read in the first, say, 100 observations.
>
> I have tried the code, which works:
>
> use mydata if ( _N<100 )
>
> However, evidently, this code goes through ALL 3 million observations to
> evaluate the expression in parentheses, which can be very time consuming
> (and sort of defeats the purpose).  Is there a way to only read the
> first 100 observations without having to evaluate the entire dataset?
>
> Perhaps some application of the "set obs 100"?  But I have not been
> successful.
>
> Thank you.
> -Mark
First, that is not officially valid syntax, though it is accepted. I 
find that it gets you 0 observations, though it does read through the 
whole file.
You probably mean _n (little n), rather than _N. (I suppose _N is . 
during the loading process, so _N <100 is false).

Officially correct syntax is,
use if _n <100 using mydata

or, better yet,
use in 1/99 using mydata

(This latter syntax is much more efficient.)

But in any case, my experience has been that it always reads through 
the whole file. And you can tell it's dong that if you have 3000000 
observations. The reason is that, in the file structure, there are 
some important elements that come after the data (values labels, I 
believe, for example), so there is a reason to have to read the whole 
file.  At least that's how it's been as far as I know; I don't know 
if they've changed the file structure in that regard in version 10.

I may have written to Stata Corp. about this some time in the past; 
if I had my way, there would either...
 be nothing after the end of the data segment, or
 be some way to jump directly to the part of the file that lies 
after the data.
(The latter idea may or may not work, depending on file-system issues.)
In either case, I would want it to not read the whole file if you 
asked for an initial subset.

But as things stand now, we are stuck with this behavior.

The only thing you can do is, if you plan to experiment on a small 
segment of the file (and want to load it many times), load a small 
segment and save it under a different name. Thus, you go through the 
lengthy process just once.

use in 1/99 using mydata
save mydata_shortversion

Later...
use mydata_shortversion
-- should load quickly.

Hope this helps.
--David

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/