--- title: Title keywords: fastai sidebar: home_sidebar nb_path: "nbs/experiments__auto.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}

auto[source]

auto(config_dict:dict, Y_df:DataFrame, X_df:DataFrame, S_df:DataFrame, loss_function_val:callable, loss_functions_test:dict, forecast_horizon:int, ts_in_val:int, ts_in_test:int, return_forecasts:bool=False, return_model:bool=True, test_auto:bool=False, verbose:bool=False)

Auto hyperparameter tuning function.

Parameters

config_dict: Dict Dictionary with configuration. Keys should be name of models. For each model specify the hyperparameter space (None will use default suggested space), hyperopt steps and timeout. Y_df: pd.DataFrame Target time series with columns ['unique_id', 'ds', 'y']. X_df: pd.DataFrame Exogenous time series with columns ['unique_id', 'ds', 'y']. S_df: pd.DataFrame Static exogenous variables with columns ['unique_id', 'ds']. and static variables. loss_function_val: function Loss function used for validation. loss_functions_test: Dictionary Loss functions used for test evaluation. (function name: string, function: fun) forecast_horizon: int Forecast horizon ts_in_val: int Number of timestamps in validation. ts_in_test: int Number of timestamps in test. return_forecasts: bool If true return forecast on test. return_model: bool If true return model. test_auto: bool If true, will only run one training step and hyperopt iteration for each model. For testing purposes, to ensure your pipeline will finish running, without waiting. verbose: If true, will print summary of dataset, model and training.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

instantiate_space[source]

instantiate_space(model, n_time_out, n_series, n_x, n_s, frequency, test)

{% endraw %} {% raw %}
{% endraw %}

AutoNF example

The forecasting task we selected is to predict the number of patients with influenza-like illnesses from the US CDC dataset, the dataset contains 7 target variables, and has 966 weeks of history.

We will be creating point forecasts with N-BEATS, N-HiTS and RNN models. The predictive features will be the autoregressive features. More information on the dataset can be found in the N-HiTS paper.

Table of Contents

  1. Installing NeuralForecast Library
  2. Data Loading and Processing
  3. Define Hyperparameter Space
  4. Hyperparameter Tuning
  5. Evaluate Results

1. Installing Neuralforecast library

You can install the released version of NeuralForecast from the Python package index with:

{% raw %}
#!pip install neuralforecast
#!pip install matplotlib
{% endraw %} {% raw %}
import matplotlib.pyplot as plt

from neuralforecast.data.datasets.long_horizon import LongHorizon
{% endraw %}

2. Data Loading and Processing

For this example we keep 10% of the observations as validation and use the latest 20% of the observations as the test set. To do so we use the sample_mask and declare the windows that will be used to train, and validate the model.

{% raw %}
Y_df, _, _ = LongHorizon.load(directory='./', group='ILI')
Y_df.head()
unique_id ds y
0 % WEIGHTED ILI 2002-01-01 -0.421499
1 % WEIGHTED ILI 2002-01-08 -0.331239
2 % WEIGHTED ILI 2002-01-15 -0.342763
3 % WEIGHTED ILI 2002-01-22 -0.199782
4 % WEIGHTED ILI 2002-01-29 -0.218426
{% endraw %} {% raw %}
n_series = len(Y_df.unique_id.unique())
n_time = len(Y_df.ds.unique()) # dataset is balanced

ts_in_test = 193
ts_in_val = 97

print('n_time', n_time)
print('n_series', n_series)
print('ts_in_test', ts_in_test)
print('ts_in_val', ts_in_val)
n_time 966
n_series 7
ts_in_test 193
ts_in_val 97
{% endraw %} {% raw %}
#                'AGE 5-24', 'ILITOTAL', 'NUM. OF PROVIDERS', 'OT']
y_plot = Y_df[Y_df.unique_id=='% WEIGHTED ILI'].y.values
x_plot = pd.to_datetime(Y_df[Y_df.unique_id=='% WEIGHTED ILI'].ds).values

plt.plot(x_plot, y_plot)
plt.axvline(x_plot[n_time-ts_in_val-ts_in_test], color='black', linestyle='-.')
plt.axvline(x_plot[n_time-ts_in_test], color='black', linestyle='-.')
plt.ylabel('Weighted ILI [ratio]')
plt.xlabel('Date')
plt.grid()
plt.show()
plt.close()
{% endraw %}

3. Define Hyperparameter Space

A temporal train-evaluation split procedure allows us to estimate the model’s generalization performance on future data unseen by the model. We use the train set to optimize the model parameters, and the validation and test sets to evaluate the accuracy of the model’s predictions.

In this case we set the space to None, that implicitly uses the predefined model space, but the space can be specified as a dictionary following the conventions of the Hyperopt package.

{% raw %}
config_dict = {'nbeats':
                       {'space': None, # Use default
                        'hyperopt_steps': 5,
                        'timeout': 60*1
                       },
               'nhits':
                       {'space': None,  # Use default
                        'hyperopt_steps': 5,
                        'timeout': 60*1
                       },
                'rnn':
                       {'space': None,  # Use default
                        'hyperopt_steps': 5,
                        'timeout': 60*1
                       }
              }
{% endraw %}

4. Hyperparameter Tuning

A temporal train-validation-test (676,97,193) split procedure allows us to estimate the model’s generalization performance on future data unseen by the model. We use the train set to optimize the model parameters, and the validation and test sets to evaluate the accuracy of the model’s predictions.

{% raw %}
forecast_horizon = 24
best_model, results = auto(config_dict=config_dict,
                           Y_df=Y_df, X_df=None, S_df=None,
                           loss_function_val=nf.losses.numpy.mae, 
                           loss_functions_test={'mae':nf.losses.numpy.mae,
                                                'mse':nf.losses.numpy.mse},
                           forecast_horizon=forecast_horizon, ts_in_val=ts_in_val, ts_in_test=ts_in_test,
                           return_forecasts=True, return_model=True,
                           test_auto=True,
                           verbose=False)
INFO:hyperopt.tpe:build_posterior_wrapper took 0.010785 seconds
INFO:hyperopt.tpe:TPE using 0 trials
WARNING: test_auto=True, MODELS WILL NOT BE TRAINED PROPERLY!
INFO:pytorch_lightning.utilities.distributed:GPU available: False, used: False
INFO:pytorch_lightning.utilities.distributed:TPU available: False, using: 0 TPU cores
INFO:pytorch_lightning.utilities.distributed:IPU available: False, using: 0 IPUs
/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."
/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."
/Users/kingtzolivares/Desktop/neuralforecast/neuralforecast/data/tsloader.py:47: UserWarning: This class wraps the pytorch `DataLoader` with a special collate function. If you want to use yours simply use `DataLoader`. Removing collate_fn
  'This class wraps the pytorch `DataLoader` with a '
/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, predict_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."
INFO:hyperopt.tpe:build_posterior_wrapper took 0.017805 seconds
INFO:hyperopt.tpe:TPE using 0 trials
INFO:pytorch_lightning.utilities.distributed:GPU available: False, used: False
INFO:pytorch_lightning.utilities.distributed:TPU available: False, using: 0 TPU cores
INFO:pytorch_lightning.utilities.distributed:IPU available: False, using: 0 IPUs
/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=linear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode)
/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/torch/nn/functional.py:652: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /tmp/pip-req-build-up3bu5e5/c10/core/TensorImpl.h:1156.)
  return torch.max_pool1d(input, kernel_size, stride, padding, dilation, ceil_mode)
INFO:hyperopt.tpe:build_posterior_wrapper took 0.009869 seconds
INFO:hyperopt.tpe:TPE using 0 trials
INFO:pytorch_lightning.utilities.distributed:GPU available: False, used: False
INFO:pytorch_lightning.utilities.distributed:TPU available: False, using: 0 TPU cores
INFO:pytorch_lightning.utilities.distributed:IPU available: False, using: 0 IPUs
{% endraw %} {% raw %}
time   = results['nbeats']['optimization_times']
losses = results['nbeats']['optimization_losses']
plt.plot(time, losses)
plt.xlabel('segs')
plt.ylabel('val loss')
Text(0, 0.5, 'val loss')
{% endraw %}

5. Evaluate Results

Here we wrangle the numpy predictions to evaluate and plot the predictions.

{% raw %}
y_hat_nhits  = results['nhits']['y_hat']#.reshape(n_series, forecast_horizon, ts_in_test)
y_hat_nbeats = results['nbeats']['y_hat']#.reshape(n_series, forecast_horizon, ts_in_test)
y_hat_rnn    = results['rnn']['y_hat']#.reshape(n_series, forecast_horizon, ts_in_test)
y_true       = results['nbeats']['y_true']#.reshape(forecast_horizon,n_series, -1)

print('\n Original Shapes')
print('1. y_hat_nhits.shape', y_hat_nhits.shape)
print('1. y_hat_nbeats.shape', y_hat_nbeats.shape)
print('1. y_hat_rnn.shape', y_hat_rnn.shape)
print('1. y_true.shape', y_true.shape)

y_hat_nbeats = results['nbeats']['y_hat'].reshape((n_series,
                                                   170, forecast_horizon))
y_hat_nhits = results['nhits']['y_hat'].reshape((n_series,
                                                 170, forecast_horizon))
y_true = results['nbeats']['y_true'].reshape((n_series,
                                              170, forecast_horizon))

print('\n Wrangled Shapes')
print('2. y_hat_nhits.shape', y_hat_nhits.shape)
print('2. y_hat_nbeats.shape', y_hat_nbeats.shape)
print('2. y_hat_rnn.shape', y_hat_rnn.shape)
print('2. y_true.shape', y_true.shape)
 Original Shapes
1. y_hat_nhits.shape (1190, 24)
1. y_hat_nbeats.shape (1190, 24)
1. y_hat_rnn.shape (7, 8, 24)
1. y_true.shape (1190, 24)

 Wrangled Shapes
2. y_hat_nhits.shape (7, 170, 24)
2. y_hat_nbeats.shape (7, 170, 24)
2. y_hat_rnn.shape (7, 8, 24)
2. y_true.shape (7, 170, 24)
{% endraw %} {% raw %}
w_idx = 0
u_idx = 0

plt.plot(y_true[u_idx,w_idx,:], label='True Signal')
plt.plot(y_hat_nbeats[u_idx,w_idx,:], label='N-BEATS')
plt.plot(y_hat_nhits[u_idx,w_idx,:], label='N-HiTS')
#plt.plot(y_true[:,0,2], label='True')
#plt.plot(best_nbeats[::24,:].flatten(), label='N-BEATS')
#plt.plot(best_rnn[::24,:].flatten(), label='RNN')
plt.legend()
plt.show()
{% endraw %} {% raw %}
print('Y_df.unique_id.unique()', Y_df.unique_id.unique())
ver = Y_df[Y_df.unique_id=='% WEIGHTED ILI']

plt.plot(ver.y[n_time-193:n_time-193+24])
plt.ylabel('% WEIGHTED ILI')
plt.show()
Y_df.unique_id.unique() ['% WEIGHTED ILI' '%UNWEIGHTED ILI' 'AGE 0-4' 'AGE 5-24' 'ILITOTAL'
 'NUM. OF PROVIDERS' 'OT']
{% endraw %}

5. Evaluate Results

{% raw %}
best_model
NBEATS(
  (model): _NBEATS(
    (blocks): ModuleList(
      (0): _NBEATSBlock(
        (layers): Sequential(
          (0): Linear(in_features=48, out_features=128, bias=True)
          (1): ReLU()
          (2): Linear(in_features=128, out_features=128, bias=True)
          (3): ReLU()
          (4): Linear(in_features=128, out_features=72, bias=True)
        )
        (basis): IdentityBasis()
      )
      (1): _NBEATSBlock(
        (layers): Sequential(
          (0): Linear(in_features=48, out_features=128, bias=True)
          (1): ReLU()
          (2): Linear(in_features=128, out_features=128, bias=True)
          (3): ReLU()
          (4): Linear(in_features=128, out_features=72, bias=True)
        )
        (basis): IdentityBasis()
      )
      (2): _NBEATSBlock(
        (layers): Sequential(
          (0): Linear(in_features=48, out_features=128, bias=True)
          (1): ReLU()
          (2): Linear(in_features=128, out_features=128, bias=True)
          (3): ReLU()
          (4): Linear(in_features=128, out_features=72, bias=True)
        )
        (basis): IdentityBasis()
      )
    )
  )
)
{% endraw %} {% raw %}
# X_forecast_df = X_df[X_df['ds']<'2016-12-28']
{% endraw %} {% raw %}
# forecast_df = best_model.forecast(Y_df=Y_forecast_df, X_df=X_forecast_df, S_df=None, batch_size=2)
# forecast_df
{% endraw %}