Re: st: error 603 using reshape

[Eric Booth <ebooth@ppri.tamu.edu>]

<>

I think you're right that it's a memory issue. Besides increasing your memory, the best option to get around the limit might be to break your dataset up into pieces, reshape it, and then merge it together.   
 The examples below are long (no doubt others will know ways to condense this code), but show you how to break up the dataset and reshape it in pieces which (I think) will enable you to get around the memory limit issue.  You didn't mention which way you were reshaping (wide or long), so both ways are shown below:

******************!
//setup dataset//
sysuse auto, clear
recode rep78 (.=9)
egen i = group(rep78 for)
bys i: g j = _n
order i j
drop for rep78
save original.dta , replace
desc, sh  //check mem usage


**********************
**LONG TO WIDE**
**********************

//1. split dataset into smaller parts and reshape them
ds i j, not
token `"`r(varlist)'"'
**this does 2 vars at a time, you could increase it**
while "`1'"!="" {
	di "for vars:  `1' and `2'"
	u "original.dta", clear
		keep i j `1' `2'
	reshape wide `1' `2', i(i) j(j)
	save using_`1'_`2'.dta, replace
	macro shift 2  //shifts to `3' and `4', and so on
	}

//2. merge these datasets
clear
set obs 1
g i = .
save master.dta, replace emptyok
global files: dir "`pwd'" files "using*", respectcase
di `"$files"'
token `"$files"'
while "`1'"!="" {
	u master.dta, clear
	merge 1:1 i  using "`1'"
		drop if _m==1
		drop _m
	desc, sh  //check mem usage
	save master.dta, replace
	cap erase "`1'"  //you don't need these
	macro shift
	}
save reshaped_wide.dta, replace	
desc, sh  //check mem usage


**********************
**WIDE TO LONG**
**********************
u reshaped_wide.dta, clear
desc, sh  //check mem usage

//1. split dataset into smaller parts and reshape them
ds i , not
local vars `r(varlist)'
di `"`vars'"'
**remove numbers and duplicates from list**
numlist  "0(1)9"
local nonum `vars'
foreach n in   `r(numlist)'  {
   local nonum: subinstr local nonum "`n'" "", all
	}
	local nonum:list uniq local(nonum)
di `"`nonum'"'  //this is our varlist for reshaping


token `"`nonum'"'
while "`1'"!="" {
	di "for stubs:  `1' and `2'"
	u "reshaped_wide.dta", clear
		keep i  `1'*  `2'*
	reshape long `1' `2', i(i) j(j)
	save forappend_`1'_`2'.dta, replace
	macro shift 2  //shifts to `3' and `4', and so on
	}

//2. merge these datasets
clear
set obs 1
g i = .
g j = .
save master2.dta, replace emptyok
global files: dir "`pwd'" files "forappend*", respectcase
di `"$files"'
token `"$files"'
while "`1'"!="" {
	u master2.dta, clear
	merge 1:1 i j using "`1'"
		drop if _m==1
		drop _m	
		desc, sh  //check mem usage
	save master2.dta, replace
	cap erase "`1'"   //you don't need these
	macro shift
	}
ds i j, not
local vl `r(varlist)'
local vl: subinstr local vl " " ", ", all
di "`vl'"
drop if mi(`vl')
save reshaped_long.dta, replace	
desc, sh  //check mem usage

//compare original and reshape long data//
cf _all using original.dta, a v 
******************!
~ Eric

__
Eric A. Booth
Public Policy Research Institute
Texas A&M University
ebooth@ppri.tamu.edu
Office: +979.845.6754



On Aug 16, 2010, at 8:53 AM, Harry Comber wrote:

> I get an intermittent error 603 when using the reshape wide command in Windows XP. This seems more likely to happen if the dataset is larger, but sometimes the command will run if I try it again on the same data. I am guessing that this may be happening because the system runs out of memory, but I have no more memory available (600Mb). Is there any way around this? The dataset has 355K obs and is about 12Mb, with 5 "j" values. I have compressed the data and reduced the number of variables to a minimum.
> Harry Comber
> 
> National Cancer Registry (www.ncri.ie)
> Building 6800
> Cork Airport Business Park
> Kinsale Road
> Cork.
> Tel  +353 21 4318014
> Fax +353 21 4318016
> Email: h.comber@ncri.ie
> ------------------------------------------------------------------------------- 
> The contents of this email are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorised designee, you may not copy or use it, or disclose it to anyone else. If you have received it in error please notify us immediately and then destroy it. The NCRI does not guarantee the security of any information electronically transmitted and is not
> liable if the information contained in this communication is not a proper and complete record of the message as transmitted by the sender or for any delay in its receipt.
>  
> 
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/




*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/