--- title: Autoformer keywords: fastai sidebar: home_sidebar summary: "API details." description: "API details." nb_path: "nbs/models_transformer__autoformer.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %}

Autoformer model wrapper

{% raw %}

class Autoformer[source]

Autoformer(seq_len:int, label_len:int, pred_len:int, output_attention:bool, enc_in:int, dec_in:int, d_model:int, c_out:int, embed:str, freq:str, dropout:float, factor:float, n_heads:int, d_ff:int, moving_avg:int, activation:str, e_layers:int, d_layers:int, loss_train:str, loss_valid:str, loss_hypar:float, learning_rate:float, lr_decay:float, weight_decay:float, lr_decay_step_size:int, random_seed:int) :: LightningModule

Hooks to be used in LightningModule.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

Autoformer.forecast[source]

Autoformer.forecast(Y_df:DataFrame, X_df:DataFrame=None, S_df:DataFrame=None, trainer:Trainer=None)

Method for forecasting self.n_time_out periods after last timestamp of Y_df.

Parameters

Y_df: pd.DataFrame Dataframe with target time-series data, needs 'unique_id','ds' and 'y' columns. X_df: pd.DataFrame Dataframe with exogenous time-series data, needs 'unique_id' and 'ds' columns. Note that 'unique_id' and 'ds' must match Y_df plus the forecasting horizon. S_df: pd.DataFrame Dataframe with static data, needs 'unique_id' column. bath_size: int Batch size for forecasting. trainer: pl.Trainer Trainer object for model training and evaluation.

Returns

forecast_df: pd.DataFrame Dataframe with forecasts.

{% endraw %} {% raw %}
{% endraw %}

Autoformer Usage Example

Load Data

{% raw %}
from neuralforecast.data.datasets.long_horizon import LongHorizon

Y_df, X_df, S_df = LongHorizon.load(directory='./data', group='ETTm2')
Y_df = Y_df.reset_index(drop=True)
Y_df.loc[Y_df['unique_id']=='OT','y'] = Y_df[Y_df['unique_id']=='OT']['y'] + 100 #To obseve differences
{% endraw %} {% raw %}
Y_df.head()
unique_id ds y
0 HUFL 2016-07-01 00:00:00 -0.041413
1 HUFL 2016-07-01 00:15:00 -0.185467
2 HUFL 2016-07-01 00:30:00 -0.257495
3 HUFL 2016-07-01 00:45:00 -0.577510
4 HUFL 2016-07-01 01:00:00 -0.385501
{% endraw %} {% raw %}
X_df.head()
unique_id ds ex_1 ex_2 ex_3 ex_4
0 HUFL 2016-07-01 00:00:00 -0.500000 0.166667 -0.5 -0.00137
1 HUFL 2016-07-01 00:15:00 -0.500000 0.166667 -0.5 -0.00137
2 HUFL 2016-07-01 00:30:00 -0.500000 0.166667 -0.5 -0.00137
3 HUFL 2016-07-01 00:45:00 -0.500000 0.166667 -0.5 -0.00137
4 HUFL 2016-07-01 01:00:00 -0.456522 0.166667 -0.5 -0.00137
{% endraw %} {% raw %}
f_cols = X_df.drop(columns=['unique_id', 'ds']).columns.to_list()
{% endraw %}

Declare Model and Data Parameters

{% raw %}
mc_model = {}

mc_model['seq_len'] = 96
mc_model['label_len'] = 48
mc_model['pred_len'] = 96
mc_model['output_attention'] = False
mc_model['enc_in'] = 7
mc_model['dec_in'] = 7
mc_model['d_model'] = 512
mc_model['c_out'] = 7
mc_model['embed'] = 'timeF'
mc_model['freq'] = 'h'
mc_model['dropout'] = 0.05
mc_model['factor'] = 1
mc_model['n_heads'] = 8
mc_model['d_ff'] = 2_048
mc_model['moving_avg'] = 25 
mc_model['activation'] = 'gelu'
mc_model['e_layers'] = 2 
mc_model['d_layers'] = 1
mc_model['loss_train'] = 'MAE'
mc_model['loss_hypar'] = 0.5
mc_model['loss_valid'] = 'MAE'
mc_model['learning_rate'] = 0.001
mc_model['lr_decay'] = 0.5
mc_model['weight_decay'] = 0.
mc_model['lr_decay_step_size'] = 2
mc_model['random_seed'] = 1

# Dataset parameters
mc_data = {}
mc_data['mode'] = 'iterate_windows'
mc_data['n_time_in'] = mc_model['seq_len']
mc_data['n_time_out'] = mc_model['pred_len']
mc_data['batch_size'] = 2
mc_data['normalizer_y'] = None
mc_data['normalizer_x'] = None
mc_data['max_epochs'] = None
mc_data['max_steps'] = 1
mc_data['early_stop_patience'] = 20

len_val = 11_520
len_test = 11_520
{% endraw %}

Instantiate Loaders and Model

{% raw %}
from neuralforecast.data.tsdataset import IterateWindowsDataset
from neuralforecast.experiments.utils import create_datasets
from torch.utils.data import DataLoader


train_dataset, val_dataset, test_dataset, scaler_y = create_datasets(mc=mc_data,
                                                                     S_df=None, 
                                                                     Y_df=Y_df, X_df=X_df,
                                                                     f_cols=f_cols,
                                                                     ds_in_val=len_val,
                                                                     ds_in_test=len_test)

train_loader = DataLoader(dataset=train_dataset,
                          batch_size=int(mc_data['batch_size']),
                          shuffle=True,
                          drop_last=True)

val_loader = DataLoader(dataset=val_dataset,
                        batch_size=int(mc_data['batch_size']),
                        shuffle=False)

test_loader = DataLoader(dataset=test_dataset,
                         batch_size=int(mc_data['batch_size']),
                         shuffle=False)
INFO:root:Train Validation splits

INFO:root:                                        ds                     
                                       min                  max
unique_id sample_mask                                          
HUFL      0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
HULL      0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
LUFL      0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
LULL      0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
MUFL      0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
MULL      0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
OT        0            2017-06-26 00:00:00  2018-02-20 23:45:00
          1            2016-07-01 00:00:00  2017-06-25 23:45:00
INFO:root:
Total data 			403200 time stamps 
Available percentage=100.0, 	403200 time stamps 
Insample  percentage=60.0, 	241920 time stamps 
Outsample percentage=40.0, 	161280 time stamps 

/Users/cchallu/NIXTLA/neuralforecast/neuralforecast/data/tsdataset.py:208: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  X.drop(['unique_id', 'ds'], 1, inplace=True)
INFO:root:Train Validation splits

INFO:root:                                        ds                     
                                       min                  max
unique_id sample_mask                                          
HUFL      0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
HULL      0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
LUFL      0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
LULL      0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
MUFL      0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
MULL      0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
OT        0            2016-07-01 00:00:00  2018-02-20 23:45:00
          1            2017-06-26 00:00:00  2017-10-23 23:45:00
INFO:root:
Total data 			403200 time stamps 
Available percentage=100.0, 	403200 time stamps 
Insample  percentage=20.0, 	80640 time stamps 
Outsample percentage=80.0, 	322560 time stamps 

/Users/cchallu/NIXTLA/neuralforecast/neuralforecast/data/tsdataset.py:208: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  X.drop(['unique_id', 'ds'], 1, inplace=True)
INFO:root:Train Validation splits

INFO:root:                                        ds                     
                                       min                  max
unique_id sample_mask                                          
HUFL      0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
HULL      0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
LUFL      0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
LULL      0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
MUFL      0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
MULL      0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
OT        0            2016-07-01 00:00:00  2017-10-23 23:45:00
          1            2017-10-24 00:00:00  2018-02-20 23:45:00
INFO:root:
Total data 			403200 time stamps 
Available percentage=100.0, 	403200 time stamps 
Insample  percentage=20.0, 	80640 time stamps 
Outsample percentage=80.0, 	322560 time stamps 

/Users/cchallu/NIXTLA/neuralforecast/neuralforecast/data/tsdataset.py:208: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  X.drop(['unique_id', 'ds'], 1, inplace=True)
{% endraw %} {% raw %}
model = Autoformer(**mc_model)
{% endraw %}

Train Model

{% raw %}
early_stopping = pl.callbacks.EarlyStopping(monitor='val_loss', 
                                            min_delta=1e-4, 
                                            patience=mc_data['early_stop_patience'],
                                            verbose=False,
                                            mode="min")

trainer = pl.Trainer(max_epochs=mc_data['max_epochs'], 
                     max_steps=mc_data['max_steps'],
                     gradient_clip_val=1.0,
                     progress_bar_refresh_rate=10, 
                     check_val_every_n_epoch=1,
                     num_sanity_val_steps=1,
                     val_check_interval=1,
                     limit_val_batches=1,
                     callbacks=[early_stopping])

trainer.fit(model, train_loader, val_loader)
/Users/cchallu/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:91: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=10)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.
  f"Setting `Trainer(progress_bar_refresh_rate={progress_bar_refresh_rate})` is deprecated in v1.5 and"
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs

  | Name  | Type        | Params
--------------------------------------
0 | model | _Autoformer | 10.5 M
--------------------------------------
10.5 M    Trainable params
0         Non-trainable params
10.5 M    Total params
42.144    Total estimated model params size (MB)
Validation sanity check:   0%|          | 0/1 [00:00<?, ?it/s]
/Users/cchallu/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 12 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."
                                                                      
/Users/cchallu/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 12 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."
Epoch 0:   0%|          | 20/34368 [00:01<44:01, 13.00it/s, loss=0.59, v_num=122, train_loss_step=0.590, val_loss=0.556, train_loss_epoch=0.590]
{% endraw %}

Make Predictions

{% raw %}
# print("outputs[0][0].shape", outputs[0][0].shape)
# print("outputs[0][1].shape", outputs[0][1].shape)
# print("outputs[0][2].shape", outputs[0][2].shape)
{% endraw %}

Forecast

{% raw %}
Y_forecast_df = Y_df[Y_df['ds']<'2017-10-24']
Y_forecast_df = Y_forecast_df.reset_index(drop=True)
Y_forecast_df.tail()
unique_id ds y
322555 OT 2017-10-23 22:45:00 99.424266
322556 OT 2017-10-23 23:00:00 99.424266
322557 OT 2017-10-23 23:15:00 99.405312
322558 OT 2017-10-23 23:30:00 99.386359
322559 OT 2017-10-23 23:45:00 99.367361
{% endraw %} {% raw %}
X_forecast_df = X_df[X_df['ds']<'2017-10-25']
X_forecast_df = X_forecast_df.reset_index(drop=True)
X_forecast_df['ds'] = pd.to_datetime(X_forecast_df['ds'])
X_forecast_df.tail()
unique_id ds ex_1 ex_2 ex_3 ex_4
323227 OT 2017-10-24 22:45:00 0.456522 -0.333333 0.266667 0.310959
323228 OT 2017-10-24 23:00:00 0.500000 -0.333333 0.266667 0.310959
323229 OT 2017-10-24 23:15:00 0.500000 -0.333333 0.266667 0.310959
323230 OT 2017-10-24 23:30:00 0.500000 -0.333333 0.266667 0.310959
323231 OT 2017-10-24 23:45:00 0.500000 -0.333333 0.266667 0.310959
{% endraw %} {% raw %}
forecast_df = model.forecast(Y_df=Y_forecast_df, X_df=X_forecast_df, S_df=S_df)
INFO:root:Train Validation splits

INFO:root:                              ds                    
                             min                 max
unique_id sample_mask                               
HUFL      0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
HULL      0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
LUFL      0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
LULL      0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
MUFL      0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
MULL      0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
OT        0           2016-07-01 2017-10-23 23:45:00
          1           2017-10-24 2017-10-24 23:45:00
INFO:root:
Total data 			323232 time stamps 
Available percentage=100.0, 	323232 time stamps 
Insample  percentage=0.21, 	672 time stamps 
Outsample percentage=99.79, 	322560 time stamps 

/Users/cchallu/NIXTLA/neuralforecast/neuralforecast/data/tsdataset.py:208: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  X.drop(['unique_id', 'ds'], 1, inplace=True)
/Users/cchallu/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:91: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=1)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.
  f"Setting `Trainer(progress_bar_refresh_rate={progress_bar_refresh_rate})` is deprecated in v1.5 and"
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
/Users/cchallu/NIXTLA/neuralforecast/neuralforecast/data/tsloader.py:47: UserWarning: This class wraps the pytorch `DataLoader` with a special collate function. If you want to use yours simply use `DataLoader`. Removing collate_fn
  'This class wraps the pytorch `DataLoader` with a '
/Users/cchallu/opt/anaconda3/envs/neuralforecast/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py:133: UserWarning: The dataloader, predict_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 12 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  f"The dataloader, {name}, does not have many workers which may be a bottleneck."
Predicting: 100%|██████████| 1/1 [00:00<00:00, 10.96it/s]
{% endraw %} {% raw %}
forecast_df
unique_id ds y
0 HUFL 2017-10-24 00:00:00 -0.567900
1 HUFL 2017-10-24 00:15:00 -0.477651
2 HUFL 2017-10-24 00:30:00 -0.877063
3 HUFL 2017-10-24 00:45:00 -1.533763
4 HUFL 2017-10-24 01:00:00 -2.264436
... ... ... ...
667 OT 2017-10-24 22:45:00 99.919777
668 OT 2017-10-24 23:00:00 99.876175
669 OT 2017-10-24 23:15:00 99.591690
670 OT 2017-10-24 23:30:00 99.588852
671 OT 2017-10-24 23:45:00 99.578384

672 rows × 3 columns

{% endraw %}