Hi again.
Thank you for the quick reply Sergiy!
There is no problem running the code you gave me. I belive that the spatwmat-command is creating something temporarily that takes a lot of memory. I have only been using Stata for a couple of months so I have never looked into a source code before - how do I do that?
Does anyone else have experience in using the spatwmat-command on "large" (more than 800 observations) datasets?
Best Regards
/Lina Jonsson
> -----Ursprungligt meddelande-----
> Fr�n: [email protected]
> [mailto:[email protected]] F�r Sergiy Radyakin
> Skickat: den 26 september 2007 12:34
> Till: [email protected]
> �mne: Re: st: SV: Re: Memory problems when creating a spatial
> weight matrix
>
> Hi Lina,
>
> may be I am missing something, but a matrix 500x500 at 4
> bytes per cell occupies 1MB of memory. At 8 bytes per cell
> 2MB respectively. How does this relate to the 500/900MB of
> memory limit? As I understand, your dataset is 800
> observations and -spatwmat- produces a matrix 800x800.
>
> Could you run the following and tell us what happens?
>
> clear
> set mem 100m <-- or whatever your maximum is
> set matsize 800
> matrix Z=J(800,800,0)
> matrix dir
> matrix A=J(800,800,0)
> matrix dir
> matrix B=J(800,800,0)
> matrix dir
> matrix C=J(800,800,0)
> matrix dir
> use "mydata" <--- whatever your dataset is
>
> If it works fine, than it is not a memory issue, but rather
> you'd have to look into the source code of -spatwmat- to find
> if the memory is used there efficiently (e.g. it does not
> generate 800 temporary matrices).
>
> From the first look, it might be relatively easy to port the
> code into matlab. However then you need matlab, why not Stata
> SE? If you don't have constraints with software -- buy Stata
> 10 SE and this might resolve the issue. If you do have
> constraints, than what are they (which software is available
> to you)? There might be just a handful of people working with
> specialized commands like this, so please explain us what
> this command does in plain English.
>
> Best regards,
> Sergiy Radyakin
>
>
>
>
>
> On 9/26/07, Lina Jonsson <[email protected]> wrote:
> > Dear statalister.
> >
> > Sorry I haven�t had time to follow up my question until now.
> >
> > The memory problem occurs only when using the command for
> creating the spatial weight matrix (spatwmat). The dataset
> I�m using consists of only 28 variables - all numerical and
> dummies (0,1). According to an earlier question concering
> memory problems when using the spatwmat-command; "-spatwmat-
> will reset the number of observations to (N/2)*N-1" which
> means about 320 000 if I�m using a dataset with 800
> observations. The advice that is given in the former question
> is to set memory to a higher value. That is the advice that
> also is given for my error message "no room to add more
> variables". But the problem is that I cannot set memory to
> more then 940m.
> >
> > I am using Windows XP and have just increased my RAM-memory
> to 2 GB which had no effect at all on the maximum memory I
> can set within Stata. As I understand it; the hotfix Michel
> mentioned is only for Stata 10 and I`m using Intercooled Stata 9.2.
> >
> > Is anyone having any suggestions on how this problem can be
> solved or is it not possible to use the spatwmat-command on
> datasets with 800 observation using "a normal computer". In
> fact I whould like to use up to 1738 observations but for
> that I whould need to switch to Stata SE instead of
> Intercooled Stata due to the fact that Intercooled Stata can
> set matsize to a maximum of 800. But there is no point in
> changing to Stata SE if the available memory sets the limit
> to less than 800 observations...
> >
> > If Stata can�t handle spatial weight matrices for large
> datasets, do any of you have any suggestions of other
> statistical software that can?
> >
> > Best regards.
> >
> > /Lina Jonsson
> >
> >
> >
> >
> > > Fr�n: [email protected]
> > > [mailto:[email protected]] F�r Michael Blasnik
> > > Skickat: den 21 september 2007 14:49
> > > Till: [email protected]
> > > �mne: st: Re: Memory problems when creating a spatial
> weight matrix
> > >
> > > You need to give us more information about your dataset,
> operating
> > > system, etc.
> > > The 940m should be plenty of room for a 500 observation
> dataset and
> > > an 800x800 matrix. Can you describe your dataset in more
> detail?
> > > Do you have hundreds of long string variables? Is there
> a way for
> > > you to encode any long strings and compress other variables? In
> > > terms of accessing more memory -- what OS are you using? If MS
> > > Windows XP, have you tried the memory hotfix described here
> > > http://www.stata.com/support/faqs/win/winmemory.html?
> > >
> > > Michael Blasnik
> > >
> > > ----- Original Message -----
> > > From: "Lina Jonsson" <[email protected]>
> > > To: <[email protected]>
> > > Sent: Friday, September 21, 2007 5:25 AM
> > > Subject: st: Memory problems when creating a spatial weight matrix
> > >
> > >
> > > > Dear statalisters.
> > > >
> > > > I am working with a hedonic price study where I whould like
> > > to check for
> > > > and correct for spatial autocorrelation. For this I
> whould like to
> > > > create a spatial weight matrix based on the x- and
> y-coordinates
> > > > for each house using the spatwmat-command in spdep. Since I
> > > have Intercooled
> > > > Stata the maximum matsize that I can set is 800, which
> > > means that I can
> > > > use no more than 800 observations. But since I cannot set
> > > memory to more
> > > > than 940m, the actual numbers of observations that I can use is
> > > > less than 500. This is to few.
> > > >
> > > > After reading some old questions concerning memory problems
> > > I realize
> > > > that this might be a problem that is hard to solve.
> > > > Does any of you have any suggestions on how I can increase
> > > the memory in
> > > > Stata. Can increasing the RAM-memory on my computer
> help? Or is it
> > > > unrealistic to think that it will be possible to create a
> > > matrix whith
> > > > the spatwmat-command that will be almost 8000x800 on a
> "standard"
> > > > computer? Should I change to some other software and what
> > > in that case?
> > > >
> > > > I am greatful for any suggestions!
> > > >
> > > > /Lina Jonsson
> > >
> > > *
> > > * For searches and help try:
> > > * http://www.stata.com/support/faqs/res/findit.html
> > > * http://www.stata.com/support/statalist/faq
> > > * http://www.ats.ucla.edu/stat/stata/
> > >
> >
> > *
> > * For searches and help try:
> > * http://www.stata.com/support/faqs/res/findit.html
> > * http://www.stata.com/support/statalist/faq
> > * http://www.ats.ucla.edu/stat/stata/
> >
>
> *
> * For searches and help try:
> * http://www.stata.com/support/faqs/res/findit.html
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
>
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
