To evaluate the performance of our models, we need some assessment statistics. Mean squared error, or MSE, calculates the square of the differences between the predicted value and the true value at each time point. These values are averaged over all time points to get the mean squared error. We often then find the root mean squared error, or RMSE, by taking the root of the MSE, to get a more interpretable statistic. Another popular assessment metric for forecasts is the mean absolute percent error, or MAPE, which looks at the percent difference between the predicted value and the true value at each time point. These percent differences are then averaged over the entire time series. There are several other assessment metrics used in different contexts, but all of them involve summing over differences between predicted values and true values of the time series. When we use these assessment statistics to judge the quality of our time series models, it’s important to determine whether we’re looking at training performance or validation performance. The ideas behind training and validation data presented earlier in the course carry forward here, but they have different names. The in-sample forecasts are forecasts on parts of the time series that were used to fit the models. These in-sample forecasts are like the training data. The information in this part of the data was used to generate the model, so performance assessments will be optimistically biased. Out-of-sample forecasts are on parts of the time series that were not used to fit the models. These out-of-sample forecasts are like the validation data. They represent data that the model has not seen before and can be used to honestly evaluate model performance. Just as it is crucial to use validation data to assess and select our predictive models, we must use out-of-sample forecasts to assess and select our forecasting models. The automatic time series modeling functionality is part of the timeData action set. It will fit a few different time series models and select the best one and report the result to the user. The best model is selected by comparing the root mean squared error on the out-of-sample holdout forecast. The automatic forecasting methods choose between exponential smoothing models and ARIMA models, and they even include exogenous variables in the ARIMA models. In the following sections, we discuss exponential smoothing models and ARIMAX models to understand what results we can get from the automatic forecasting method. At the end of this lesson, we use deep learning models that we discussed previously to model time series data as well.