Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
| From | Brendan Halpin <brendan.halpin@ul.ie> |
| To | statalist@hsphsun2.harvard.edu |
| Subject | Re: st: Importing subset of a pipe delimited textfile |
| Date | Wed, 17 Oct 2012 21:51:29 +0100 |
On Wed, Oct 17 2012, Rob Shaw wrote: > I have a very large (around 4Gb) text file that has been pipe > delimited. It won't all fit in memory so I want to process it in > parts. > > For fixed datasets I would use infile with the in 1/10000000 option > then 10000001/2000000 etc. However, this dataset has been pipe > delimited so I would need to use insheet, but insheet doesn't seem to > permit the "in" option. If you're on Mac, Linux or Solaris, the well-worn unix text utilities will deal with this neatly. In particular -split- will create subfile (and sed will change | to \t, but filefilter copes with that) Something like split --numeric-suffixes --lines=10000 bigfile.raw should create multiple files containing 10000 lines, with a numeric suffix (bigfile.raw01 and so on, I think), which you can then handle cleanly in Stata. If you're on that other OS, there's always cygwin. Brendan -- Brendan Halpin, Department of Sociology, University of Limerick, Ireland Tel: w +353-61-213147 f +353-61-202569 h +353-61-338562; Room F1-009 x 3147 mailto:brendan.halpin@ul.ie ULSociology on Facebook: http://on.fb.me/fjIK9t http://teaching.sociology.ul.ie/bhalpin/wordpress twitter:@ULSociology * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/