RE: st: ARIMA models - determining how much final model predicts actual trend

[Kerri Coomber <Kerri.Coomber@cancervic.org.au>]

Thank you Nick for your prompt reply.  I had read of this type of analytic approach (i.e., estimating how much of the final model/covariates in the model explains the actual trend) being used in a few of journal articles where the authors used time series analysis.  I was interested in trying to replicate their approach, and to see if it was a valid 'next step' after specifying a final ARIMA model   I had not considered a couple of the cautions you have outlined, so thank you for your insights, they are very helpful.  Using you advice, I will compare the models with the covariates with the covariates to the one without; keeping in mind the number of reservations to such an estimate.
  

-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Wednesday, 20 June 2012 7:51 AM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: ARIMA models - determining how much final model predicts actual trend

I don't know what the "observed trend" means independently of what is predicted by a time series model. You can construct any descriptive statistic you like based on e.g. correlation between observed and predicted, or its square, but I'd surround that with cautions. Some or all of this may naturally be obvious to you, but here goes.

1. -arima- in Stata is estimated by maximum likelihood. So, on a purist view no statistic has strong meaning unless it is equivalent one-to-one to the likelihood.

2. At a wild guess you want something akin to what is otherwise more familiar to you, from say multiple regression or analysis of variance.
But in those cases something like R-square has some meaning in so far as (e.g.) least squares can be interpreted as maximising R-square. See
#1 above, again.

3. These single-number measures tend to be used as propaganda weapons (Look at how good my model is!) whereas tougher-minded evaluations tend to be based on whether a model fitted to part of a time series successfully predicts the rest of the data, whether the fitted model matches independently posited ideas about the generating process, and so on.

4. Standard technical and philosophical reservations about whether curve-fitting amounts to any kind of "explanation".

5. No single-number measure can capture much pattern, e.g. in what the model misses, whether the fit is equally good or poor at all times, etc.

6. This can stand for plenty of other reservations that just don't happen to spring to mind right now.

That said, comparing a model with covariates with one without sounds like very good advice to me.

Nick

On Tue, Jun 19, 2012 at 10:22 PM, Kerri Coomber <Kerri.Coomber@cancervic.org.au> wrote:

> I have been conducting a time series analysis using an arima model. I have specified my final model and would now like to determine how much this model (and therefore the covariates in the model) predict the actual downward trend in the data.  Basically I would like to be able to state that "Using the final model to predict the outcome variable we determined that it explained XX amount of change in the outcome, or XX% of the observed trend"
>
> I have attempted this by using the predict y command.  I then calculated the difference in the predicted outcome (between the start and end if the series) and compared this to the actual change.
>
> It has also been suggested to me that I compare the predicted trend using the full model to the predicted trend using a model that does not have the covariates.
>
> Has anyone conducted such an analysis, or have any suggestions as to what may be the most appropriate approach?

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/