RE: st: A bug in egen and gen?

["Liao, Junlin" <junlin-liao@uiowa.edu>]

I expanded my experiment to include one more variable that contains a string ("Experiment for testing Stata Options"). The results are the same:

No compress, with option float: 39,064KB
No compress, with option double: 42,970KB
Compressed with either option: 39,064KB

Basically float=long in terms of storage. The option for numeric variables has no impact on string variables - as expected.

Junlin Liao
Surgery Finance, 1422 JCP
Phone: (319) 356-2588


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Sarah Edgington
Sent: Thursday, February 17, 2011 2:01 PM
To: statalist@hsphsun2.harvard.edu
Subject: RE: st: A bug in egen and gen?

Junlin,
Have you tried this experiment with something other than a large integer?  I think Nick's point was that you only regain the space using compress if you're dealing exclusively with integers.  You've demonstrated that compressing a double variable in the case where all observations are integers gets you the same storage size as if you'd started out in float.
Is the same true if all observations are not integers?
-Sarah

-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Liao, Junlin
Sent: Thursday, February 17, 2011 11:53 AM
To: statalist@hsphsun2.harvard.edu
Subject: RE: st: A bug in egen and gen?

Nick,

I had experimented with Stata in terms of storage. Here are the results:

I generate one variable with a single value of 83085733 for 1000,000 times.
The different sizes are
                                 Original file after -compress-
Float                      3907KB
3907KB (Long)
Double                 7813KB
3907KB (Long)

I can see that if the variable is of type double, it requires twice as much storage space comparing to float. The storage space for float is as much as for long. There is no difference after the files are compressed to the final appropriate data type. Therefore, my recommendation for Stata to use double as default calculation and finally select the appropriate type to store data is sensible.

It is also desirable to simply set type to double and compress whenever saving data.

Thanks,

Junlin


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Liao, Junlin
Sent: Thursday, February 17, 2011 1:29 PM
To: statalist@hsphsun2.harvard.edu
Subject: RE: st: A bug in egen and gen?

I'm confused here now. Isn't the type of variables determines storage spaces? I'll do some experiments to see your point here. Your attention and quick responses are greatly appreciated.

Junlin


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Thursday, February 17, 2011 1:23 PM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: A bug in egen and gen?

You're saying, in effect, that nearly doubling storage would not typically bite users. That will be true in some cases but not all.

There is no need to wonder. The help for -save- says there is no such option. But -compress- before -save- is naturally a very good choice and you could program your own wrapper for -save- that always did it.

Here is a sketch:

program jlsave
version 8
compress
save `0'
end

But if you do what you just said you wanted to do, -set type double-, using
-compress- is not going to give you back more than a fraction of the extra storage you spend. The fraction will depend on how much you deal with strings, always integer variables, etc.

On Thu, Feb 17, 2011 at 7:11 PM, Liao, Junlin <junlin-liao@uiowa.edu> wrote:

Storage wouldn't be a problem if we perform -compress- command regularly.
I'm wondering if Stata can let you select an option whenever it saves data, it compresses. It will surely be handy to solve this problem.

Nick Cox

> In practice, if StataCorp always warned you of everything that could
> bite
you, the help would be much, much longer.
>
> Your last suggestion would typically leave -double-s in place unless
> it so
happened that the result was integers in every observation. To see why, study Bill Gould's recent postings on the StataCorp blog.
> That would on average nearly double your storage. If you don't mind,
> you
might as follow Stas' suggestion and -set type double-.
>
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
________________________________

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
________________________________

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
________________________________

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/