Dear all:
The computer specification which I found out this strange result is (windows
cpu E6600/ram 8G). I noticed that STATA 10 (64bit/MP2) in my computer seems
to be spending more time in executing the same program than STATA 9 (64bit).
That is why I posted this message to see if anyone shares similar
experiences.
Nonetheless, given that other users suggested running tests more seriously,
I have used two other computers (windows, cpu 6600/ram 4g) to test the
program; both yielded similar results. Our data and memory specification as
well as the new outputs are listed in following:
1. data size used for analysis is around 1.75G
2. use the same program (see the following reported log file ), and I let
Stata to run the same program with different memory size: 2.2G, 2.4G, 2.6G,
2.8G, and 3.0G
3. when I set memory=2.2G, Stata 9 can do our program in 66.454second,
however, Stata 10 can not open the data set due to the memory constraint
(this is part I do not understand)
4. when I gradually increase the memory from 2.4, 2.6, 2.8 to 3.0, and the
time to finish our program for Stata 9 and 10 is: (stata 9): 69.063, 73.094,
77.844, 81.5 (stata 10): 57.969, 49.719, 54.735, 58.781 (I also do not
understand why STATA9 takes more time in executing the program when a larger
memory is assigned)
5. We actually found that Stata 10 is faster than Stata 9 if the memory is
assigned large enough.
So far I cannot be sure whether it is the problem of my computer or STATA10
is slower in that particular hardware specification or the combination of
both. I have tried to find another computer with similar specifications to
see if I can replicate the result produced in my computer, and will report
them once new results are available.
=======log file for stata 9 =============
----------------------------------------------------------------------------
-----------------------------------
log: e:\nhi\he\log\test_9.log
log type: text
opened on: 1 Sep 2007, 11:57:30
.
. timer on 1
. set mem 2200m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.771M
set memory 2200M max. data space 2,200.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,203.025M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,138,549,695 (7.3% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A033091100" | order_code==
> "A028034100" | order_code=="A033675100" | order_code=="A034016100" |
order_code=="A034927100" | order_code
> =="A036396100" | order_code=="A036756100" | order_code=="A036857100" |
order_code=="A037211100" | order_co
> de=="A038262100" | order_code=="A038494100" | order_code=="A039559100"
> |
order_code=="A039575100" | order_
> code=="A041251100" | order_code=="A042830100" |
> order_code=="A043972100" |
order_code=="B015133100" | orde
> r_code=="B018857100" | order_code=="B020841100" | order_code=="B018965100"
| order_code=="B015133100" | or
> der_code=="B018857100" )
.
.
. timer off 1
. timer list 1
1: 66.45 / 1 = 66.4540
.
.
. clear
. timer on 2
. set mem 2400m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.771M
set memory 2400M max. data space 2,400.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,403.025M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,138,549,695 (15.0% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A033091100" | order_code==
> "A028034100" | order_code=="A033675100" | order_code=="A034016100" |
order_code=="A034927100" | order_code
> =="A036396100" | order_code=="A036756100" | order_code=="A036857100" |
order_code=="A037211100" | order_co
> de=="A038262100" | order_code=="A038494100" | order_code=="A039559100"
> |
order_code=="A039575100" | order_
> code=="A041251100" | order_code=="A042830100" |
> order_code=="A043972100" |
order_code=="B015133100" | orde
> r_code=="B018857100" | order_code=="B020841100" | order_code=="B018965100"
| order_code=="B015133100" | or
> der_code=="B018857100" )
.
.
. timer off 2
. timer list 2
2: 69.06 / 1 = 69.0630
.
.
. clear
. timer on 3
. set mem 2600m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.771M
set memory 2600M max. data space 2,600.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,603.025M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,138,549,695 (21.6% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A033091100" | order_code==
> "A028034100" | order_code=="A033675100" | order_code=="A034016100" |
order_code=="A034927100" | order_code
> =="A036396100" | order_code=="A036756100" | order_code=="A036857100" |
order_code=="A037211100" | order_co
> de=="A038262100" | order_code=="A038494100" | order_code=="A039559100"
> |
order_code=="A039575100" | order_
> code=="A041251100" | order_code=="A042830100" |
> order_code=="A043972100" |
order_code=="B015133100" | orde
> r_code=="B018857100" | order_code=="B020841100" | order_code=="B018965100"
| order_code=="B015133100" | or
> der_code=="B018857100" )
.
.
. timer off 3
. timer list 3
3: 73.09 / 1 = 73.0940
.
. clear
. timer on 4
. set mem 2800m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.771M
set memory 2800M max. data space 2,800.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,803.025M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,138,549,695 (27.2% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A033091100" | order_code==
> "A028034100" | order_code=="A033675100" | order_code=="A034016100" |
order_code=="A034927100" | order_code
> =="A036396100" | order_code=="A036756100" | order_code=="A036857100" |
order_code=="A037211100" | order_co
> de=="A038262100" | order_code=="A038494100" | order_code=="A039559100"
> |
order_code=="A039575100" | order_
> code=="A041251100" | order_code=="A042830100" |
> order_code=="A043972100" |
order_code=="B015133100" | orde
> r_code=="B018857100" | order_code=="B020841100" | order_code=="B018965100"
| order_code=="B015133100" | or
> der_code=="B018857100" )
.
.
. timer off 4
. timer list 4
4: 77.84 / 1 = 77.8440
.
.
. clear
. timer on 5
. set mem 3000m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.771M
set memory 3000M max. data space 3,000.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
3,003.025M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,138,549,695 (32.0% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A033091100" | order_code==
> "A028034100" | order_code=="A033675100" | order_code=="A034016100" |
order_code=="A034927100" | order_code
> =="A036396100" | order_code=="A036756100" | order_code=="A036857100" |
order_code=="A037211100" | order_co
> de=="A038262100" | order_code=="A038494100" | order_code=="A039559100"
> |
order_code=="A039575100" | order_
> code=="A041251100" | order_code=="A042830100" |
> order_code=="A043972100" |
order_code=="B015133100" | orde
> r_code=="B018857100" | order_code=="B020841100" | order_code=="B018965100"
| order_code=="B015133100" | or
> der_code=="B018857100" )
.
.
. timer off 5
. timer list 5
5: 81.50 / 1 = 81.5000
.
.
.
.
. log close
log: e:\nhi\he\log\test_9.log
log type: text
closed on: 1 Sep 2007, 12:03:38
----------------------------------------------------------------------------
--------------------------------
======================log file for stata 10
===* part I
----------------------------------------------------------------------------
--------------
log: e:\nhi\he\log\test_10.log
log type: text
opened on: 1 Sep 2007, 12:09:40
.
. timer on 1
. set mem 2200m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.947M
set memory 2200M max. data space 2,200.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,203.201M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
no room to add more observations
An attempt was made to increase the number of observations beyond what
is currently
possible. You have the following alternatives:
1. Store your variables more efficiently; see help compress. (Think
of Stata's data
area as the area of a rectangle; Stata can trade off width and
length.)
2. Drop some variables or observations; see help drop.
3. Increase the amount of memory allocated to the data area using the
set memory
command; see help memory.
r(901);
end of do-file
r(901);
. do "C:\DOCUME~1\jtsheu\LOCALS~1\Temp\STD00000000.tmp"
. cd e:\nhi\he
e:\nhi\he
.
. log using log\test_10a.log, replace
log file already open
r(604);
end of do-file
r(604);
. log close
log: e:\nhi\he\log\test_10.log
log type: text
closed on: 1 Sep 2007, 12:10:06
----------------------------------------------------------------------------
--------------
====log file for stata 10 another part==============
----------------------------------------------------------------------------
--------------
log: e:\nhi\he\log\test_10a.log
log type: text
opened on: 1 Sep 2007, 12:10:08
.
.
. clear
. timer on 2
. set mem 2400m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.947M
set memory 2400M max. data space 2,400.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,403.201M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,283,536,115 (15.0% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A0330911
> 00" | order_code=="A028034100" | order_code=="A033675100" |
order_code=="A034016100" | o
> rder_code=="A034927100" | order_code=="A036396100" |
order_code=="A036756100" | order_co
> de=="A036857100" | order_code=="A037211100" | order_code=="A038262100"
> |
order_code=="A0
> 38494100" | order_code=="A039559100" | order_code=="A039575100" |
order_code=="A04125110
> 0" | order_code=="A042830100" | order_code=="A043972100" |
order_code=="B015133100" | or
> der_code=="B018857100" | order_code=="B020841100" |
order_code=="B018965100" | order_cod
> e=="B015133100" | order_code=="B018857100" )
.
.
. timer off 2
. timer list 2
2: 57.97 / 1 = 57.9690
.
.
. clear
. timer on 3
. set mem 2600m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.947M
set memory 2600M max. data space 2,600.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,603.201M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,283,536,115 (21.6% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A0330911
> 00" | order_code=="A028034100" | order_code=="A033675100" |
order_code=="A034016100" | o
> rder_code=="A034927100" | order_code=="A036396100" |
order_code=="A036756100" | order_co
> de=="A036857100" | order_code=="A037211100" | order_code=="A038262100"
> |
order_code=="A0
> 38494100" | order_code=="A039559100" | order_code=="A039575100" |
order_code=="A04125110
> 0" | order_code=="A042830100" | order_code=="A043972100" |
order_code=="B015133100" | or
> der_code=="B018857100" | order_code=="B020841100" |
order_code=="B018965100" | order_cod
> e=="B015133100" | order_code=="B018857100" )
.
.
. timer off 3
. timer list 3
3: 49.72 / 1 = 49.7190
.
. clear
. timer on 4
. set mem 2800m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.947M
set memory 2800M max. data space 2,800.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
2,803.201M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,283,536,115 (27.2% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A0330911
> 00" | order_code=="A028034100" | order_code=="A033675100" |
order_code=="A034016100" | o
> rder_code=="A034927100" | order_code=="A036396100" |
order_code=="A036756100" | order_co
> de=="A036857100" | order_code=="A037211100" | order_code=="A038262100"
> |
order_code=="A0
> 38494100" | order_code=="A039559100" | order_code=="A039575100" |
order_code=="A04125110
> 0" | order_code=="A042830100" | order_code=="A043972100" |
order_code=="B015133100" | or
> der_code=="B018857100" | order_code=="B020841100" |
order_code=="B018965100" | order_cod
> e=="B015133100" | order_code=="B018857100" )
.
.
. timer off 4
. timer list 4
4: 54.73 / 1 = 54.7350
.
.
. clear
. timer on 5
. set mem 3000m
Current memory allocation
current memory usage
settable value description (1M = 1024k)
--------------------------------------------------------------------
set maxvar 5000 max. variables allowed 1.947M
set memory 3000M max. data space 3,000.000M
set matsize 400 max. RHS vars in models 1.254M
-----------
3,003.201M
. set more off
. cd e:\nhi\he
e:\nhi\he
.
.
. *create data
. use data\he_do2001.dta, clear
. d
Contains data from data\he_do2001.dta
obs: 36,246,605
vars: 13 24 Mar 2007 22:31
size: 2,283,536,115 (32.0% of memory free)
----------------------------------------------------------------------------
---
storage display value
variable name type format label variable label
----------------------------------------------------------------------------
---
fee_ym str6 %9s �O�Φ~��
appl_type str1 %9s �ӳ����O
hosp_id str10 %10s ��ƾ��c�N��
case_type str1 %9s �ץ����
seq_no float %9.0g �y����
order_seq_no int %9.0g ��O�Ǹ�
order_type str1 %9s ��O���O
order_code str12 %12s ��O�N�X
rate_type float %9.0g ��I����
order_qty float %9.0g ��O�ƶq
order_price float %9.0g ��O���
order_amt float %9.0g ��O���B
appl_date int %dCY/N/D �ӳ����
----------------------------------------------------------------------------
---
Sorted by:
.
. *generate indicators of plavix (stent)
. gen byte plavix_tmp=(order_code=="B022932100")
. gen byte ticlo_tmp=(order_code=="A031596100" |
order_code=="A0330911
> 00" | order_code=="A028034100" | order_code=="A033675100" |
order_code=="A034016100" | o
> rder_code=="A034927100" | order_code=="A036396100" |
order_code=="A036756100" | order_co
> de=="A036857100" | order_code=="A037211100" | order_code=="A038262100"
> |
order_code=="A0
> 38494100" | order_code=="A039559100" | order_code=="A039575100" |
order_code=="A04125110
> 0" | order_code=="A042830100" | order_code=="A043972100" |
order_code=="B015133100" | or
> der_code=="B018857100" | order_code=="B020841100" |
order_code=="B018965100" | order_cod
> e=="B015133100" | order_code=="B018857100" )
.
.
. timer off 5
. timer list 5
5: 58.78 / 1 = 58.7810
.
.
.
.
. log close
log: e:\nhi\he\log\test_10a.log
log type: text
closed on: 1 Sep 2007, 12:13:50
----------------------------------------------------------------------------
--------------
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
