To evaluate the performance of our models,

we need some assessment statistics.

Mean squared error, or MSE, calculates

the square of the differences between the predicted value

and the true value at each time point.

These values are averaged over all time points

to get the mean squared error.

We often then find the root mean squared error, or RMSE,

by taking the root of the MSE, to get

a more interpretable statistic.

Another popular assessment metric for forecasts

is the mean absolute percent error, or MAPE,

which looks at the percent difference

between the predicted value and the true value

at each time point.

These percent differences are then

averaged over the entire time series.

There are several other assessment metrics

used in different contexts, but all of them

involve summing over differences between predicted values

and true values of the time series.

When we use these assessment statistics to judge the quality

of our time series models, it’s important to determine whether

we’re looking at training performance or validation

performance.

The ideas behind training and validation data

presented earlier in the course carry forward here,

but they have different names.

The in-sample forecasts are forecasts

on parts of the time series that were used to fit the models.

These in-sample forecasts are like the training data.

The information in this part of the data

was used to generate the model, so performance assessments

will be optimistically biased.

Out-of-sample forecasts are on parts

of the time series that were not used to fit the models.

These out-of-sample forecasts are like the validation data.

They represent data that the model has not seen before

and can be used to honestly evaluate model performance.

Just as it is crucial to use validation data to assess

and select our predictive models,

we must use out-of-sample forecasts to assess and select

our forecasting models.

The automatic time series modeling

functionality is part of the timeData action set.

It will fit a few different time series models

and select the best one and report the result to the user.

The best model is selected by comparing

the root mean squared error on the out-of-sample holdout

forecast.

The automatic forecasting methods

choose between exponential smoothing models and ARIMA

models, and they even include exogenous variables

in the ARIMA models.

In the following sections, we discuss exponential smoothing

models and ARIMAX models to understand

what results we can get from the automatic forecasting method.

At the end of this lesson, we use deep learning models

that we discussed previously to model time series data as well.