RE: st: -gllamm- vs -meglm-

[Timothy Mak <tshmak@hku.hk>]

Hi Dan, 

Single random intercept logistic models is not what the original post was about. I think Stata should be able to do that quickly also. 

The original post involves calculations of conditional likelihood. I must admit I'm intrigued why groups involving >1000 observations caused a problem in the calculations. But based on my limited understanding of conditional likelihood, it's probably to do with having a sum of some 1000+ elements, and perhaps rounding errors are causing some of the sums to be negative. But if anyone knows more, please comment! 

Tim

PS I do have one complaint over Stata's handling of large data. I too have worked with GBs of data. One thing I've found is that if I have a huge dataset in the memory, then even if I want to do calculations in, say, a subset of 100 observations, the calculations are very very much slower than if I only had that subset in memory. Sometimes, it's even faster to -preserve- the data, drop the unnecessary observations, do the calculations, and -restore-. This, I believe, is due to the numerous calls to ado-subroutines that involve things like -marksample touse- and -... if `touse'-, etc. 

PPS Another thing is that -reshape- pretty much grinds to a halt with large datasets. Otherwise, it's probably my favourite command! 


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Daniel Waxman
Sent: 03 July 2013 10:12
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: -gllamm- vs -meglm-

containing such large groups..."

3,000 observations in a group isn't what it used to be!  There are
many of us who routinely work with gigabytes of data, and it would be
helpful if Stata's documentation made it clear from the outset that
some estimation routines are not meant for us, if they are not.

Somehow SAS's glimmix routine is able to fit at least single random
intercept logistic models on huge datasets relatively quickly (and it
does so while using very little memory).    To the extent that the
Stata development team is willing to comment on a competing product,
I'd be interested on their take on whether Stata will be able to do
the same any time soon.

Thanks!
Dan
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/