Stata/MP4 9.2 running on Linux/amd64 with four CPUs.
In the process of tracking down some resource issues on a Linux/amd64 I
have found some slightly strange behaviour with Stata/MP and, under
certain circumstances, it fails to free up RAM used during an ODBC load.
Carefully monitoring the memory usage for the following DO file, the
memory usage appears at right-hand side (virtual/resident), which is the
memory usage reportedly used by the Stata process at the completion of
the command:
set memory 1100m 1138M / 25M
odbc load, exec("select * from viewname") dsn(datasource)
rises to 10.6G / 9.7G
during load
then drops back to 1152M / 845M
after load
xi: regress [stuff] 1164M / 846M
clear 1164M / 846M
odbc load, exec("select * from viewname") dsn(datasource)
rises to 10.6G / 10.0G
and *stays* there!
However, if one ensures that only one CPU is used, then the RAM *is*
freed up at the end of the second 'odbc load':
(Close, then restart Stata)
set procs_use 1
set memory 1100m 1138M / 25M
odbc load, exec("select * from viewname") dsn(datasource)
rises to 10.6G / 9.7G
during load
then drops back to 1152M / 845M
after load
xi: regress [stuff] 1164M / 846M
clear 1164M / 846M
odbc load, exec("select * from viewname") dsn(datasource)
rises to 10.6G / 10.0G
during load
then drops back to 1152M / 846M
Note that after the second 'odbc load', this time the RAM is released.
Further investigation reveals that if the 'xi: regress' command is
replaced with a 'tab', the RAM is freed up at the end. I'm guessing
that the distinction is that 'regress' performs some parallel execution
and that 'tab' does not.
I understand that ODBC caches results from the database query in RAM
during load, but should release this back to the system when the data
load is complete. It does do this in single-processor mode, but not if
something multi-processor has been run.
We noticed this problem when we ran into resource utilisation issues
following an upgrade from Stata/SE to Stata/MP a few weeks ago.
Basically, with Stata/ODBC not freeing up the RAM, the system started
swapping an awful lot.
I've started suggesting that Stata users ensure that they 'exit' before
starting a second ODBC/load session, since that always frees up memory,
but this is rather annoying, since staff routinely keep a single Stata
session running almost continuously. And some staff have large numbers
of existing DO files which make use of multiple 'odbc load's.
Is this a Stata bug? Is this an ODBC bug? Something else?
The problematic symptoms are certainly specific to the multi-processor
part of Stata. This never happened previously under Stata/SE, but
started happening with Stata/MP. The fact that the problem goes away if
you specify 'set procs_use 1' confirms this.
Cheers,
Dave.
--
Dave Ewart
[email protected]
Computing Manager, Cancer Epidemiology Unit
Cancer Research UK / Oxford University
PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370
Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc
N 51.7518, W 1.2016
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
