--- title: Giacomini-White Test keywords: fastai sidebar: home_sidebar nb_path: "nbs/losses__gw_test.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %}

Giacomini-White Conditional Predictive Ability Test

{% raw %}

GW_CPA_test[source]

GW_CPA_test(loss1:ndarray, loss2:ndarray, tau:int, alpha:float=0.05, conditional:bool=False, verbose:bool=True)

The one-sided Giacomini-White Conditional Predictive Ability Test (GW), allows to assess which model provides better predictions.

The GW test examines the null hypothesis of equal forecast errors for a pair of models's predictions $\hat{\mathbf{y}}^{A}_{\tau}$ and $\hat{\mathbf{y}}^{B}_{\tau}$ measured by the MAE or L1 norm, conditioned on the available information to that moment $\mathcal{F}_{\tau-1}$.

$$ \Delta^{A,B}_{\tau} = ||\mathbf{y}_{\tau} - \hat{\mathbf{y}}^{A}_{\tau}||_{1} - ||\mathbf{y}_{\tau} - \hat{\mathbf{y}}^{B}_{\tau}||_{1} $$

In practice, the $ \mathcal{F}_{\tau-1} $ is replaced with a constant and lags of the error difference $ \mathcal{F}_{\tau-1} = [\mathbb{1} \;|\; \Delta^{A,B}_{\tau - 1} ] $ and the test is performed using a linear regression with a Wald-like test.

$$ \Delta^{A,B}_{\tau} = \boldsymbol{\beta}^{\intercal} \mathcal{F}_{\tau-1} + \epsilon_{\tau} $$$$ H_{0}: \mathbb{E} \left[ \Delta^{A,B}_{\tau} \;|\; \mathcal{F}_{\tau-1} \right] \equiv \mathbb{E} \left[ \boldsymbol{\beta} \;|\; \mathcal{F}_{\tau-1} \right] = \mathbb{0} $$

The Diebold-Mariano test widely used in the forecasting literature, can be easily recovered by setting, when the conditional information considered is only the constant variable, one recovers the original DB test. Compared with the DM or other unconditional tests, the GW test is valid under general assumptions such as heterogeneity rather than stationarity of data.

Parameters
----------
loss1:
    numpy array. losses of model 1
loss2:
    numpy array. losses of model 2
tau: int
    the past information treated as 'available' for the test.
alpha: float
    level of significance for the test.
conditional: boolean.
    True if conditional (DM test), False if unconditional (GW test).
verbose: boolean.
    True if prints of test are needed.

Returns
-------
test_stat: test statistic of the conditional predictive ability test
crit_val: critical value of the chi-square test for a 5% confidence level
p-vals: (k,) p-value of the test

References
----------
[1] Giacomini, R., & White, H., Tests of conditional predictive ability. Econometrica, 74
URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-0262.2006.00718.

[2] Diebold, F., & Mariano, R. (2002). Comparing predictive accuracy.
Journal of Business & Economic Statistics, 20 , 134–44.
URL: https://www.sas.upenn.edu/~fdiebold/papers/paper68/pa.dm.pdf.
{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %}

Giacomini-White Conditional p-values

{% raw %}

get_GW_test_pvals[source]

get_GW_test_pvals(y:ndarray, y_hat:ndarray, horizon:int, tau:int, conditional:bool, alpha:float=0.05, verbose:bool=False)

Function to calculate model-pair-wise GW-Test p-values.

Parameters
----------
y: numpy array
    flat array with actual test values
y_hat: numpy array
    matrix with predicted values
horizon: int
    the multi horizon for which the predictions were created,
    the test is performed against over the mean differences
    in said multi horizon losses.
tau: int
    the past information treated as 'available' for the test.
conditional: boolean,
    True if conditional (DM test), False if unconditional (GW test).
alpha: float
    level of significance for the test.
verbose: boolean.
    True for partial test results.

Returns
-------
p_vals: (n_models, n_models)
        symmetric numpy array with the model-pair-wise p-values.
{% endraw %} {% raw %}
{% endraw %} {% raw %}

plot_GW_test_pvals[source]

plot_GW_test_pvals(pvals:ndarray, labels:List[str], title:str)

Function to plot model-pair-wise GW-Test p-values.

Parameters
----------
p_vals: (n_models, n_models)
    symmetric numpy array with the model-pair-wise p-values.
labels: string list
    list of model names
title: string
    title of the plot
{% endraw %} {% raw %}
{% endraw %}

Examples

1. GW with ModelB prefered to ModelA.

{% raw %}
lossA = np.random.randint(low=1, high=10, size=(11,1))
lossB = lossA.copy() - lossA/10

plt.plot(lossA, label='Model A', color="#78ACA8")
plt.plot(lossB, label='Model B', color="#E3A39A")
plt.ylabel('Errors')
plt.xlabel('Time')

plt.legend()
plt.grid()
plt.show()
plt.close()

test_stat, crit_val, p_val = GW_CPA_test(loss1=lossA, loss2=lossB, tau=2, conditional=True)
GW Conditional test:

Forecast horizon: 2, Nominal Risk Level: 0.05
Test-statistic: [28.22091328] (+)
Critical value: 5.991464547107979
p-value: [7.44571733e-07]

{% endraw %}

2. GW with ModelA prefered to ModelB.

{% raw %}
lossA = np.random.randint(low=1, high=10, size=(11,1))
lossB = lossA.copy() + lossA/10

plt.plot(lossA, label='Model A', color="#78ACA8")
plt.plot(lossB, label='Model B', color="#E3A39A")
plt.ylabel('Errors')
plt.xlabel('Time')

plt.legend()
plt.grid()
plt.show()
plt.close()

test_stat, crit_val, p_val = GW_CPA_test(loss1=lossA, loss2=lossB, tau=2, conditional=True)
GW Conditional test:

Forecast horizon: 2, Nominal Risk Level: 0.05
Test-statistic: [-100.0144055] (-)
Critical value: 5.991464547107979
p-value: [1.]

{% endraw %}

3. GW pairwise models' comparison

{% raw %}
{% endraw %} {% raw %}
y = np.random.randint(low=1, high=10, size=(100*24)) 

# Five different models with augmenting error variance
eps0 = np.random.normal(loc=0.0, scale=1, size=(100*24,))
eps1 = np.random.normal(loc=0.0, scale=2, size=(100*24,))
eps2 = np.random.normal(loc=0.0, scale=3, size=(100*24,))
eps3 = np.random.normal(loc=0.0, scale=3, size=(100*24,))
eps4 = np.random.normal(loc=0.0, scale=3, size=(100*24,))

y_hat = np.repeat(y[:,None], repeats=5, axis=1)
y_hat[:,0] = y_hat[:,0] + eps0
y_hat[:,1] = y_hat[:,1] + eps1
y_hat[:,2] = y_hat[:,2] + eps2
y_hat[:,3] = y_hat[:,3] + eps3
y_hat[:,4] = y_hat[:,4] + eps4

model_names = ['Model1', 'Model2', 'Model3', 'Model4', 'Model5']
pvals = get_GW_test_pvals(y=y, y_hat=y_hat, horizon=24, tau=1,
                          conditional=True, alpha=0.05, verbose=False)

print("Model 1 is prefered over all the other models")
print("Model 5 is non-significantly prefered over Model 4")
plot_GW_test_pvals(pvals=pvals, labels=model_names, title='GW_test')
Model 1 is prefered over all the other models
Model 5 is non-significantly prefered over Model 4
{% endraw %}