Forecast Model Selection

Forecast model selection tests are statistical methods that determine whether the accuracy difference between two competing forecasting models is statistically significant or merely due to random chance. Without these tests, you might deploy a more complex model that appears better on a single test set but offers no real improvement.

---

Diebold-Mariano Test

The Diebold-Mariano test is a statistical test that evaluates whether two forecasting models have significantly different predictive accuracy. It compares the loss differential between two sets of forecasts and tests whether the mean difference is statistically distinguishable from zero.

Parameters

Parameter	Type	Required	Default	Description
actual	DOUBLE	Yes	-	Actual values
forecast1	DOUBLE	Yes	-	First forecast
forecast2	DOUBLE	Yes	-	Second forecast
options	MAP	No	-	Configuration options

Options MAP:

Option	Type	Default	Description
loss	VARCHAR	mse	mse, mae, or mape
alternative	VARCHAR	two_sided	two_sided, less, greater

Output

Field	Type	Description
statistic	DOUBLE	DM test statistic
p_value	DOUBLE	p-value
better_model	INTEGER	1 or 2 (which forecast is better)

Example

SELECT anofox_stats_diebold_mariano_agg(
    actual,
    forecast1,
    forecast2,
    MAP {'loss': 'mse'}
) as result
FROM forecast_data;

---

Clark-West Test

The Clark-West test is designed for comparing nested forecasting models, where one model (restricted) is a special case of the other (unrestricted). It adjusts for the parameter estimation noise that makes the Diebold-Mariano test conservative in nested comparisons. More powerful than Diebold-Mariano when comparing nested models.

Parameters

Parameter	Type	Required	Default	Description
actual	DOUBLE	Yes	-	Actual values
forecast1	DOUBLE	Yes	-	First forecast (restricted model)
forecast2	DOUBLE	Yes	-	Second forecast (unrestricted model)

Output

Field	Type	Description
statistic	DOUBLE	CW test statistic
p_value	DOUBLE	p-value

Example

SELECT anofox_stats_clark_west_agg(
    actual,
    forecast1,
    forecast2
) as result
FROM forecast_data;

---

Choosing Between Tests

Scenario	Recommended Test
Non-nested models	Diebold-Mariano
Nested models	Clark-West
Large errors matter more	DM with loss: 'mse'
Robust to outliers	DM with loss: 'mae'
Percentage errors	DM with loss: 'mape'