Gold, Oil, NASDAQ, USD-index example¶

This notebook illustrates the toolkit's use on a real-world financial data set, using the weekly price of the gold, oil, the NASDAQ and an index of the USD.

For these four time series we fit three different multi-output Gaussian processes and show how they compare. For each model we optimize we repeat the experiment 5 times, to ensure robustness in the optimization and prediction.

Data sources:

  • Oil: Brent daily in USD from https://www.eia.gov/dnav/pet/hist/RBRTEd.htm
  • Gold: LMBA gold daily at 10:30AM in USD from https://fred.stlouisfed.org/series/GOLDAMGBD228NLBM
  • NASDAQ: adjusted closing price (daily) from https://finance.yahoo.com/quote/%5EIXIC/history?p=%5EIXIC
  • USD index: traded weighted on broad currency index from https://fred.stlouisfed.org/series/TWEXB

Please see our publication at https://doi.org/10.1109/ICASSP40776.2020.9054102 for a full description of this data set and the interpretations of our results.

In [1]:
import mogptk
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Data loading¶

We will load the four financial time series and put them in a single data set for MOGPTK. Next we filter for a specific time range, in this case four years, and agregate the daily measurements into weeks. Additionally, we transform the Y axis in order to make training more effective.

For each of the channels we also remove certain ranges so that we can evaluate the effectiveness of data imputation in those regions. Each trial will additionally remove data points at random in order to verify the consistency of the results.

In [2]:
oil     = mogptk.LoadCSV('data/gonu/brent-daily.csv', 'Date', 'Price',
                         name='Oil')
gold    = mogptk.LoadCSV('data/gonu/lmba-gold-usd-am-daily.csv', 'Date',
                         'Price', name='Gold', na_values='.')
nasdaq  = mogptk.LoadCSV('data/gonu/nasdaq.csv', 'Date', 'Adj Close',
                         name='NASDAQ')
usd     = mogptk.LoadCSV('data/gonu/TWEXB.csv', 'Date', 'Price',
                         name='USD')
dataset = mogptk.DataSet(gold, oil, nasdaq, usd)

# Remove points randomly to simulate sensor failure
np.random.seed(1)
for i, channel in enumerate(dataset):
    channel.filter('2015-01-01', '2018-12-31')
    channel.aggregate('7D')
    
    channel.transform(mogptk.TransformDetrend())
    channel.transform(mogptk.TransformNormalize())
    
    name = channel.get_name()
    if name == 'Gold':
        channel.remove_range('2016-11-15', '2017-01-01')
    if name == 'Oil':
        channel.remove_range('2018-10-05', None)
    if name == 'USD':
        channel.remove_range('2016-03-15', '2016-06-01')

dataset.plot();

We will be using the Adam optimizer, which is known to give good results using a relatively high learning rate, and we'll run five trials for each model in order to evaluate the consistency in our results.

In [3]:
n_trials = 5
Q = 3
init_method = 'BNSE'
method = 'Adam'
lr = 0.1
iters = 500

Multi-output spectral mixture kernel¶

In [4]:
mosm_models = [None] * n_trials
mosm_mae = np.zeros((n_trials,4))
mosm_rmse = np.zeros((n_trials,4))
mosm_mape = np.zeros((n_trials,4))

for n in range(n_trials):
    mosm_dataset = dataset.copy()
    mosm_dataset['Gold'].remove_randomly(pct=0.6)
    mosm_dataset['Oil'].remove_randomly(pct=0.3)
    mosm_dataset['NASDAQ'].remove_randomly(pct=0.6)
    mosm_dataset['USD'].remove_randomly(pct=0.6)
    
    print('\nTrial', n+1, 'of', n_trials)
    mosm = mogptk.MOSM(mosm_dataset, Q=Q)
    mosm.init_parameters(init_method)
    try:
        mosm.train(method=method, lr=lr, iters=iters, verbose=True, error='MSE')
    except mogptk.CholeskyException:
        mosm_mape[n,:] = 100.0
        continue
    finally:
        print('=' * 50)
    
    error = mogptk.error(mosm, per_channel=True)[0]
    mosm_mae[n,:] = np.array([item['MAE'] for item in error])
    mosm_rmse[n,:] = np.array([item['RMSE'] for item in error])
    mosm_mape[n,:] = np.array([item['MAPE'] for item in error])
    mosm_models[n] = mosm
Trial 1 of 5

Starting optimization using Adam
‣ Model: MOSM
‣ Channels: 4
‣ Parameters: 64
‣ Training points: 382
‣ Initial loss: 163.265
‣ Initial error: 33912.3

Start Adam:
    0/500   0:00:00  loss=     163.265  error=     33912.3
    5/500   0:00:00  loss=     110.908  error=     21673.1
   10/500   0:00:00  loss=     90.1259  error=     15471.9
   15/500   0:00:01  loss=     71.8853  error=     14842.6
   20/500   0:00:01  loss=     59.0471  error=     18186.3
   25/500   0:00:02  loss=     42.7907  error=     15343.8
   30/500   0:00:02  loss=     29.7533  error=     14642.8
   35/500   0:00:03  loss=      18.056  error=     15022.1
   40/500   0:00:03  loss=     7.75873  error=     15028.7
   45/500   0:00:04  loss=    -1.60781  error=     15672.9
   50/500   0:00:04  loss=    -10.1516  error=     15368.1
   55/500   0:00:05  loss=    -18.6789  error=     15152.5
   60/500   0:00:05  loss=    -23.1161  error=     15128.5
   65/500   0:00:05  loss=    -29.3066  error=     14638.3
   70/500   0:00:06  loss=    -32.2809  error=     14680.1
   75/500   0:00:06  loss=    -36.8353  error=     14588.1
   80/500   0:00:07  loss=    -40.4958  error=     14512.9
   85/500   0:00:07  loss=    -45.1421  error=     14836.4
   90/500   0:00:07  loss=     -48.476  error=       16075
   95/500   0:00:08  loss=      -45.42  error=     13883.2
  100/500   0:00:08  loss=    -37.2327  error=     13606.1
  105/500   0:00:08  loss=    -39.5822  error=       14651
  110/500   0:00:09  loss=    -47.3859  error=     13848.3
  115/500   0:00:09  loss=    -54.1082  error=     13479.8
  120/500   0:00:09  loss=    -43.3206  error=     16051.6
  125/500   0:00:10  loss=    -50.0582  error=     13415.3
  130/500   0:00:10  loss=    -54.2387  error=     13555.6
  135/500   0:00:11  loss=    -67.4962  error=     12559.7
  140/500   0:00:11  loss=     -68.624  error=     13492.8
  145/500   0:00:12  loss=    -71.5204  error=     12269.3
  150/500   0:00:12  loss=    -73.2573  error=     12959.5
  155/500   0:00:13  loss=    -77.6595  error=     12454.3
  160/500   0:00:13  loss=    -69.1319  error=     13092.1
  165/500   0:00:14  loss=    -66.8633  error=       12746
  170/500   0:00:14  loss=    -75.6747  error=     12101.3
  175/500   0:00:14  loss=    -77.0916  error=     12173.9
  180/500   0:00:15  loss=    -78.4037  error=     11794.3
  185/500   0:00:15  loss=    -79.7755  error=     11445.5
  190/500   0:00:16  loss=    -80.0256  error=     11555.9
  195/500   0:00:16  loss=    -81.3633  error=     11439.7
  200/500   0:00:17  loss=    -82.1395  error=     11475.3
  205/500   0:00:17  loss=     -83.982  error=     11356.3
  210/500   0:00:18  loss=    -85.4834  error=     11189.2
  215/500   0:00:18  loss=     -86.796  error=     11135.1
  220/500   0:00:18  loss=    -88.8722  error=     11114.7
  225/500   0:00:19  loss=    -89.7006  error=     11033.2
  230/500   0:00:19  loss=    -90.7221  error=     10940.7
  235/500   0:00:20  loss=    -91.4485  error=     10899.9
  240/500   0:00:20  loss=    -91.9957  error=     10823.6
  245/500   0:00:21  loss=    -92.8058  error=     10756.7
  250/500   0:00:21  loss=    -96.4645  error=     10712.7
  255/500   0:00:22  loss=    -100.014  error=     10722.8
  260/500   0:00:22  loss=    -103.649  error=     10925.7
  265/500   0:00:23  loss=    -106.075  error=     10763.9
  270/500   0:00:24  loss=    -106.197  error=     10804.2
  275/500   0:00:24  loss=     -96.681  error=     10817.7
  280/500   0:00:24  loss=    -96.8856  error=     11145.6
  285/500   0:00:25  loss=    -102.034  error=     10564.4
  290/500   0:00:25  loss=    -107.466  error=       11151
  295/500   0:00:25  loss=     -110.34  error=       10704
  300/500   0:00:26  loss=    -113.037  error=     10453.8
  305/500   0:00:26  loss=    -114.785  error=     10641.6
  310/500   0:00:27  loss=    -115.786  error=     10629.2
  315/500   0:00:27  loss=    -117.627  error=     10465.1
  320/500   0:00:27  loss=    -118.955  error=       10703
  325/500   0:00:28  loss=    -119.864  error=     10791.4
  330/500   0:00:28  loss=    -120.634  error=     10675.3
  335/500   0:00:29  loss=    -121.123  error=     10683.7
  340/500   0:00:30  loss=    -120.247  error=     10914.6
  345/500   0:00:30  loss=    -121.223  error=     10405.5
  350/500   0:00:31  loss=    -119.395  error=     10706.4
  355/500   0:00:31  loss=    -119.994  error=     10253.8
  360/500   0:00:32  loss=    -121.374  error=     10340.8
  365/500   0:00:32  loss=    -123.158  error=     11029.7
  370/500   0:00:32  loss=    -122.302  error=     10592.6
  375/500   0:00:33  loss=    -120.763  error=     10097.8
  380/500   0:00:33  loss=    -123.727  error=     10410.3
  385/500   0:00:33  loss=    -121.172  error=     11183.6
  390/500   0:00:34  loss=    -123.839  error=     10218.5
  395/500   0:00:34  loss=    -124.974  error=     10668.9
  400/500   0:00:35  loss=    -124.685  error=     10894.1
  405/500   0:00:35  loss=    -125.295  error=       10331
  410/500   0:00:35  loss=    -125.548  error=       10615
  415/500   0:00:36  loss=    -126.016  error=     10596.9
  420/500   0:00:36  loss=    -125.597  error=     10332.5
  425/500   0:00:36  loss=    -126.198  error=     10434.8
  430/500   0:00:37  loss=    -125.923  error=     10500.8
  435/500   0:00:37  loss=    -126.669  error=     10525.1
  440/500   0:00:37  loss=     -125.98  error=     10474.2
  445/500   0:00:38  loss=    -126.721  error=     10397.4
  450/500   0:00:38  loss=    -126.678  error=     10471.6
  455/500   0:00:38  loss=    -126.983  error=     10401.2
  460/500   0:00:39  loss=    -126.909  error=     10447.5
  465/500   0:00:39  loss=    -127.016  error=     10349.5
  470/500   0:00:39  loss=    -127.491  error=     10443.9
  475/500   0:00:40  loss=    -127.737  error=     10512.9
  480/500   0:00:40  loss=    -127.809  error=     10449.9
  485/500   0:00:40  loss=    -127.973  error=     10545.6
  490/500   0:00:41  loss=    -128.197  error=     10519.9
  495/500   0:00:41  loss=    -128.331  error=     10465.7
  500/500   0:00:42  loss=     -128.59  error=       10498
Finished

Optimization finished in 42.277 seconds
‣ Iterations: 500
‣ Final loss: -128.59
‣ Final error:  10498
==================================================

Trial 2 of 5

Starting optimization using Adam
‣ Model: MOSM
‣ Channels: 4
‣ Parameters: 64
‣ Training points: 387
‣ Initial loss: 150.202
‣ Initial error: 24010.6

Start Adam:
    0/500   0:00:00  loss=     150.202  error=     24010.6
    5/500   0:00:00  loss=      130.43  error=       30851
   10/500   0:00:00  loss=     108.066  error=     22326.3
   15/500   0:00:01  loss=     90.2571  error=     20633.5
   20/500   0:00:01  loss=     72.4094  error=     20989.8
   25/500   0:00:01  loss=     53.6629  error=     18328.7
   30/500   0:00:02  loss=     39.7272  error=     18639.6
   35/500   0:00:02  loss=     27.7051  error=     18443.9
   40/500   0:00:02  loss=      16.256  error=     17609.9
   45/500   0:00:03  loss=     5.86293  error=     16646.4
   50/500   0:00:03  loss=    -4.41171  error=     17128.5
   55/500   0:00:04  loss=    -11.7145  error=     16759.3
   60/500   0:00:04  loss=    -18.8692  error=     16427.5
   65/500   0:00:04  loss=    -25.1322  error=     15965.7
   70/500   0:00:05  loss=    -29.9688  error=     16125.7
   75/500   0:00:05  loss=    -32.6103  error=     16155.5
   80/500   0:00:05  loss=    -36.3609  error=     14687.1
   85/500   0:00:06  loss=    -41.2311  error=       15242
   90/500   0:00:06  loss=    -45.1827  error=       14587
   95/500   0:00:06  loss=    -46.3925  error=     13985.4
  100/500   0:00:07  loss=    -48.5431  error=     14981.9
  105/500   0:00:07  loss=    -48.0231  error=     15080.9
  110/500   0:00:07  loss=    -49.3878  error=     14142.5
  115/500   0:00:08  loss=    -53.5482  error=     14952.8
  120/500   0:00:08  loss=    -51.9809  error=     14404.3
  125/500   0:00:09  loss=    -15.2558  error=     14604.3
  130/500   0:00:09  loss=    -31.0275  error=     13666.6
  135/500   0:00:10  loss=    -23.9895  error=       13865
  140/500   0:00:10  loss=     19.9609  error=       13458
  145/500   0:00:11  loss=     -14.847  error=     13892.9
  150/500   0:00:11  loss=    -37.0531  error=     13753.2
  155/500   0:00:12  loss=    -49.4418  error=     13788.5
  160/500   0:00:12  loss=    -55.3798  error=     14554.5
  165/500   0:00:12  loss=    -54.5495  error=     13971.9
  170/500   0:00:13  loss=    -57.3876  error=     13879.2
  175/500   0:00:13  loss=    -59.0939  error=       14152
  180/500   0:00:14  loss=    -60.6208  error=     14239.8
  185/500   0:00:14  loss=    -61.7762  error=     13865.3
  190/500   0:00:14  loss=    -62.6285  error=       13723
  195/500   0:00:15  loss=    -63.2577  error=     13867.4
  200/500   0:00:15  loss=    -63.8483  error=     13795.2
  205/500   0:00:16  loss=    -64.2421  error=     13804.1
  210/500   0:00:16  loss=    -64.6307  error=     13749.1
  215/500   0:00:16  loss=    -64.9133  error=     13751.9
  220/500   0:00:17  loss=    -65.2799  error=     13592.5
  225/500   0:00:17  loss=     -65.072  error=     13779.9
  230/500   0:00:18  loss=    -65.4229  error=     13905.6
  235/500   0:00:18  loss=    -65.0016  error=     14273.9
  240/500   0:00:19  loss=    -65.0753  error=     14018.9
  245/500   0:00:19  loss=    -64.8463  error=     13382.7
  250/500   0:00:20  loss=    -65.0403  error=     14014.5
  255/500   0:00:21  loss=    -64.4021  error=     14107.8
  260/500   0:00:21  loss=    -64.6466  error=     13435.6
  265/500   0:00:21  loss=    -64.0701  error=       14831
  270/500   0:00:22  loss=     -65.186  error=     14211.2
  275/500   0:00:22  loss=    -65.4259  error=     13817.1
  280/500   0:00:22  loss=    -67.1975  error=     13479.6
  285/500   0:00:23  loss=    -68.0605  error=     13773.2
  290/500   0:00:23  loss=    -68.8357  error=     13945.4
  295/500   0:00:23  loss=    -69.3174  error=     14033.6
  300/500   0:00:24  loss=    -69.4457  error=     13954.2
  305/500   0:00:24  loss=    -67.7949  error=     14073.6
  310/500   0:00:25  loss=    -69.3247  error=       13682
  315/500   0:00:25  loss=    -69.8717  error=     14532.2
  320/500   0:00:25  loss=    -70.1453  error=     14180.7
  325/500   0:00:26  loss=    -67.7351  error=     14149.2
  330/500   0:00:26  loss=    -70.0727  error=     14229.1
  335/500   0:00:26  loss=    -68.8734  error=     13784.8
  340/500   0:00:27  loss=    -69.7681  error=     13752.6
  345/500   0:00:27  loss=     -70.103  error=     14221.7
  350/500   0:00:28  loss=    -70.0177  error=     14275.8
  355/500   0:00:28  loss=     -70.215  error=     14645.1
  360/500   0:00:28  loss=    -68.9912  error=     14609.8
  365/500   0:00:29  loss=    -69.8964  error=       13583
  370/500   0:00:29  loss=    -69.6161  error=     14934.8
  375/500   0:00:29  loss=    -69.5411  error=     13954.3
  380/500   0:00:30  loss=    -66.3976  error=     13865.8
  385/500   0:00:30  loss=    -67.4308  error=     13270.7
  390/500   0:00:30  loss=    -68.7624  error=       13207
  395/500   0:00:31  loss=    -69.6367  error=     14678.6
  400/500   0:00:31  loss=    -69.4983  error=     14350.7
  405/500   0:00:31  loss=    -70.2396  error=     13662.9
  410/500   0:00:32  loss=    -67.7631  error=     14492.2
  415/500   0:00:32  loss=    -69.2666  error=       13640
  420/500   0:00:32  loss=    -69.1559  error=     13467.5
  425/500   0:00:33  loss=    -69.4313  error=     13462.6
  430/500   0:00:33  loss=     -70.422  error=       13525
  435/500   0:00:33  loss=    -70.5736  error=     14277.6
  440/500   0:00:34  loss=    -71.3117  error=     13224.1
  445/500   0:00:34  loss=    -72.1513  error=       13438
  450/500   0:00:35  loss=    -72.5131  error=     13529.9
  455/500   0:00:35  loss=    -72.7662  error=       13423
  460/500   0:00:35  loss=    -73.1027  error=     13358.8
  465/500   0:00:36  loss=    -73.3488  error=     13366.2
  470/500   0:00:36  loss=    -73.5668  error=     13355.3
  475/500   0:00:36  loss=    -73.8144  error=     13294.9
  480/500   0:00:37  loss=    -74.0435  error=     13267.9
  485/500   0:00:37  loss=     -74.273  error=     13255.6
  490/500   0:00:37  loss=    -74.5035  error=     13222.3
  495/500   0:00:37  loss=    -74.7259  error=     13193.2
  500/500   0:00:38  loss=     -74.945  error=     13186.2
Finished

Optimization finished in 38.350 seconds
‣ Iterations: 500
‣ Final loss: -74.945
‣ Final error: 13186.2
==================================================

Trial 3 of 5

Starting optimization using Adam
‣ Model: MOSM
‣ Channels: 4
‣ Parameters: 64
‣ Training points: 384
‣ Initial loss: 176.657
‣ Initial error: 29545.8

Start Adam:
    0/500   0:00:00  loss=     176.657  error=     29545.8
    5/500   0:00:00  loss=     131.056  error=     9098.88
   10/500   0:00:00  loss=     108.569  error=     8680.74
   15/500   0:00:01  loss=     94.5667  error=     9764.75
   20/500   0:00:01  loss=     72.2104  error=     7892.47
   25/500   0:00:01  loss=     58.8011  error=     8402.46
   30/500   0:00:02  loss=     41.7108  error=     8741.38
   35/500   0:00:02  loss=     26.9533  error=     8423.03
   40/500   0:00:02  loss=     16.4999  error=     8971.71
   45/500   0:00:03  loss=     3.68687  error=     8761.11
   50/500   0:00:03  loss=    -1.71638  error=     8548.32
   55/500   0:00:03  loss=     -8.9148  error=     8673.13
   60/500   0:00:04  loss=     -18.932  error=     8787.09
   65/500   0:00:04  loss=    -21.9204  error=     8695.46
   70/500   0:00:04  loss=    -28.6644  error=     8708.75
   75/500   0:00:05  loss=    -27.5663  error=     8540.42
   80/500   0:00:05  loss=    -27.8812  error=     8892.96
   85/500   0:00:06  loss=    -28.2286  error=     8642.43
   90/500   0:00:06  loss=    -35.7809  error=     8806.48
   95/500   0:00:06  loss=    -37.5346  error=     8569.09
  100/500   0:00:07  loss=    -45.4423  error=     8745.97
  105/500   0:00:07  loss=    -37.3818  error=     8526.29
  110/500   0:00:07  loss=    -44.2992  error=     8455.96
  115/500   0:00:08  loss=    -38.8073  error=       10984
  120/500   0:00:08  loss=    -38.4587  error=     8907.29
  125/500   0:00:08  loss=     -35.825  error=     9020.83
  130/500   0:00:09  loss=    -46.9402  error=     8509.65
  135/500   0:00:09  loss=    -41.0715  error=     9121.18
  140/500   0:00:09  loss=    -18.6655  error=     10614.3
  145/500   0:00:10  loss=    -37.8289  error=     7942.83
  150/500   0:00:10  loss=    -41.9811  error=     8866.56
  155/500   0:00:10  loss=    -39.7241  error=     9961.86
  160/500   0:00:11  loss=    -48.0789  error=     8378.73
  165/500   0:00:11  loss=    -50.7819  error=      8521.8
  170/500   0:00:11  loss=    -52.7483  error=     9783.59
  175/500   0:00:12  loss=    -56.6346  error=     8409.82
  180/500   0:00:12  loss=    -57.8258  error=     8434.23
  185/500   0:00:12  loss=    -60.2135  error=     8563.68
  190/500   0:00:13  loss=    -60.2968  error=     8842.66
  195/500   0:00:13  loss=     -62.259  error=     8651.35
  200/500   0:00:13  loss=    -62.7866  error=     8613.07
  205/500   0:00:14  loss=    -63.0773  error=     8706.94
  210/500   0:00:14  loss=    -59.6362  error=     8590.32
  215/500   0:00:14  loss=    -56.7951  error=     8630.46
  220/500   0:00:15  loss=    -63.7853  error=     8618.24
  225/500   0:00:15  loss=    -64.0406  error=     8456.17
  230/500   0:00:15  loss=    -65.0652  error=     8213.63
  235/500   0:00:16  loss=    -66.5334  error=     8438.94
  240/500   0:00:16  loss=    -67.2745  error=     8094.56
  245/500   0:00:16  loss=    -64.4646  error=     8511.21
  250/500   0:00:17  loss=    -65.6406  error=     8128.95
  255/500   0:00:17  loss=    -66.7158  error=     8089.85
  260/500   0:00:17  loss=    -66.7747  error=     7919.97
  265/500   0:00:18  loss=     -69.779  error=     7836.37
  270/500   0:00:18  loss=    -62.1146  error=     8054.84
  275/500   0:00:18  loss=    -65.4754  error=     7893.97
  280/500   0:00:19  loss=    -62.8015  error=     8190.31
  285/500   0:00:19  loss=     -64.922  error=     8014.98
  290/500   0:00:19  loss=    -67.8701  error=     7886.81
  295/500   0:00:20  loss=    -64.6622  error=     8028.33
  300/500   0:00:20  loss=    -67.9703  error=     7766.71
  305/500   0:00:20  loss=    -70.4228  error=     8041.96
  310/500   0:00:21  loss=    -68.6246  error=      7816.7
  315/500   0:00:21  loss=     -71.218  error=     8023.05
  320/500   0:00:22  loss=    -71.7767  error=     7730.19
  325/500   0:00:22  loss=    -71.4378  error=     7737.43
  330/500   0:00:22  loss=     -69.966  error=      7760.4
  335/500   0:00:23  loss=    -66.8778  error=     7808.52
  340/500   0:00:23  loss=    -65.2112  error=     7903.39
  345/500   0:00:23  loss=    -68.3631  error=     7929.14
  350/500   0:00:24  loss=    -69.9629  error=     8041.54
  355/500   0:00:24  loss=    -71.1498  error=     8082.31
  360/500   0:00:24  loss=    -73.0693  error=     8224.94
  365/500   0:00:25  loss=    -75.1662  error=     8550.64
  370/500   0:00:25  loss=    -77.1899  error=     8496.27
  375/500   0:00:25  loss=    -78.7719  error=     8670.16
  380/500   0:00:26  loss=     -80.113  error=     7725.44
  385/500   0:00:26  loss=    -81.0144  error=     8055.19
  390/500   0:00:26  loss=    -77.2257  error=     7687.68
  395/500   0:00:27  loss=    -77.5241  error=     7642.28
  400/500   0:00:27  loss=    -79.7965  error=     7683.92
  405/500   0:00:28  loss=    -75.6563  error=     7782.31
  410/500   0:00:28  loss=     -77.739  error=     7832.38
  415/500   0:00:28  loss=    -79.3734  error=     7849.95
  420/500   0:00:29  loss=    -83.6553  error=     7867.66
  425/500   0:00:29  loss=    -84.3665  error=     7952.46
  430/500   0:00:29  loss=    -81.4209  error=     8148.05
  435/500   0:00:30  loss=    -81.9439  error=     7972.96
  440/500   0:00:30  loss=    -82.3912  error=     7281.18
  445/500   0:00:30  loss=    -82.5817  error=     6697.16
  450/500   0:00:31  loss=    -83.9103  error=     6341.91
  455/500   0:00:31  loss=    -87.1768  error=     6225.59
  460/500   0:00:32  loss=    -83.9866  error=     6405.45
  465/500   0:00:32  loss=    -83.7573  error=     6446.51
  470/500   0:00:32  loss=     -88.464  error=     6218.61
  475/500   0:00:33  loss=    -86.9123  error=     6442.21
  480/500   0:00:33  loss=    -89.5874  error=     6726.12
  485/500   0:00:33  loss=    -89.5425  error=     7055.66
  490/500   0:00:34  loss=    -89.0532  error=     7385.82
  495/500   0:00:34  loss=    -91.5709  error=     7372.74
  500/500   0:00:34  loss=    -83.3815  error=      7828.8
Finished

Optimization finished in 34.728 seconds
‣ Iterations: 500
‣ Final loss: -83.3815
‣ Final error: 7828.8
==================================================

Trial 4 of 5

Starting optimization using Adam
‣ Model: MOSM
‣ Channels: 4
‣ Parameters: 64
‣ Training points: 385
‣ Initial loss: 152.136
‣ Initial error: 23764.1

Start Adam:
    0/500   0:00:00  loss=     152.136  error=     23764.1
    5/500   0:00:00  loss=     138.851  error=       29165
   10/500   0:00:00  loss=     111.781  error=     17983.7
   15/500   0:00:01  loss=      95.349  error=     16081.6
   20/500   0:00:01  loss=     80.5864  error=     16155.1
   25/500   0:00:01  loss=     62.6252  error=     17559.5
   30/500   0:00:02  loss=     44.6251  error=     15144.3
   35/500   0:00:02  loss=      28.382  error=     13979.4
   40/500   0:00:02  loss=     13.5742  error=     13578.5
   45/500   0:00:03  loss=     3.41461  error=     15046.9
   50/500   0:00:03  loss=    -5.89246  error=     13552.5
   55/500   0:00:03  loss=    -12.4978  error=     13445.1
   60/500   0:00:04  loss=    -19.3976  error=     13107.2
   65/500   0:00:04  loss=    -27.9147  error=     13183.8
   70/500   0:00:05  loss=    -6.42429  error=     13265.5
   75/500   0:00:05  loss=    -35.4461  error=       12672
   80/500   0:00:05  loss=     -34.675  error=     12519.9
   85/500   0:00:06  loss=    -42.1543  error=     12395.6
   90/500   0:00:06  loss=    -44.2703  error=     12148.2
   95/500   0:00:06  loss=    -49.0603  error=     12254.6
  100/500   0:00:07  loss=    -50.2282  error=     12077.6
  105/500   0:00:07  loss=    -46.5984  error=     12059.9
  110/500   0:00:07  loss=    -50.9684  error=     11042.6
  115/500   0:00:08  loss=    -51.2291  error=     10434.3
  120/500   0:00:08  loss=    -35.6606  error=     10646.7
  125/500   0:00:08  loss=    -36.7919  error=     12223.1
  130/500   0:00:09  loss=    -41.1582  error=     12896.9
  135/500   0:00:09  loss=    -42.6251  error=     12575.9
  140/500   0:00:09  loss=    -46.8342  error=     12953.5
  145/500   0:00:10  loss=    -47.8879  error=     12288.9
  150/500   0:00:10  loss=    -51.5803  error=       12901
  155/500   0:00:11  loss=    -52.8572  error=     12161.3
  160/500   0:00:11  loss=    -53.7549  error=     12284.1
  165/500   0:00:11  loss=    -54.1542  error=     12452.1
  170/500   0:00:12  loss=    -54.9421  error=     12297.2
  175/500   0:00:12  loss=    -55.3747  error=     12108.3
  180/500   0:00:12  loss=    -55.7187  error=     12380.8
  185/500   0:00:13  loss=    -49.9457  error=     12210.6
  190/500   0:00:13  loss=    -50.0343  error=     12154.6
  195/500   0:00:13  loss=     -49.513  error=     12193.6
  200/500   0:00:14  loss=    -46.5188  error=     12180.7
  205/500   0:00:14  loss=    -41.7953  error=     12134.2
  210/500   0:00:14  loss=    -47.3145  error=     12113.5
  215/500   0:00:15  loss=    -53.0313  error=     12074.3
  220/500   0:00:15  loss=    -52.1385  error=     12078.7
  225/500   0:00:15  loss=     -53.859  error=     12038.6
  230/500   0:00:16  loss=    -55.5964  error=     12004.4
  235/500   0:00:16  loss=    -55.3933  error=     11958.2
  240/500   0:00:16  loss=    -56.4387  error=       11911
  245/500   0:00:17  loss=    -56.8963  error=     11790.4
  250/500   0:00:17  loss=    -57.2528  error=     11589.1
  255/500   0:00:17  loss=    -57.7566  error=     11268.4
  260/500   0:00:18  loss=    -58.5159  error=       10901
  265/500   0:00:18  loss=     -59.276  error=     10580.2
  270/500   0:00:18  loss=     -59.874  error=     10274.5
  275/500   0:00:19  loss=    -59.8558  error=     9984.58
  280/500   0:00:19  loss=    -57.9666  error=     9376.69
  285/500   0:00:19  loss=    -56.5212  error=     9269.71
  290/500   0:00:20  loss=     -65.318  error=     9089.03
  295/500   0:00:20  loss=    -66.0403  error=     9060.54
  300/500   0:00:20  loss=    -61.3161  error=     9136.81
  305/500   0:00:21  loss=    -60.6169  error=     9149.32
  310/500   0:00:21  loss=    -64.7846  error=     9141.15
  315/500   0:00:21  loss=    -67.4919  error=     9154.43
  320/500   0:00:22  loss=    -69.0751  error=     9196.54
  325/500   0:00:22  loss=    -69.0456  error=     9191.67
  330/500   0:00:22  loss=     -69.858  error=     9190.21
  335/500   0:00:23  loss=    -70.2736  error=     9212.17
  340/500   0:00:23  loss=    -70.4989  error=     9228.97
  345/500   0:00:23  loss=    -70.4923  error=      9211.6
  350/500   0:00:24  loss=    -70.2674  error=     9228.24
  355/500   0:00:24  loss=    -69.4385  error=     9251.84
  360/500   0:00:24  loss=      -68.67  error=      9260.1
  365/500   0:00:25  loss=    -70.0123  error=     9248.15
  370/500   0:00:25  loss=    -70.2229  error=     9235.18
  375/500   0:00:25  loss=    -67.8934  error=     9215.82
  380/500   0:00:26  loss=     -70.286  error=     9221.38
  385/500   0:00:26  loss=    -70.4832  error=      9238.5
  390/500   0:00:26  loss=    -71.1434  error=     9239.65
  395/500   0:00:27  loss=    -71.2202  error=     9250.26
  400/500   0:00:27  loss=    -71.5318  error=     9239.22
  405/500   0:00:27  loss=    -71.9049  error=     9223.01
  410/500   0:00:28  loss=    -68.9252  error=     9231.63
  415/500   0:00:28  loss=    -69.6491  error=     9270.09
  420/500   0:00:28  loss=    -70.9582  error=      9234.9
  425/500   0:00:29  loss=     -70.163  error=     9277.31
  430/500   0:00:29  loss=    -70.1356  error=     9217.34
  435/500   0:00:29  loss=    -70.1617  error=     9254.67
  440/500   0:00:30  loss=    -67.4226  error=     9263.42
  445/500   0:00:30  loss=    -70.1693  error=     9246.56
  450/500   0:00:30  loss=    -72.3304  error=     9248.48
  455/500   0:00:31  loss=    -71.0205  error=     9229.65
  460/500   0:00:31  loss=    -72.3741  error=     9223.15
  465/500   0:00:31  loss=    -73.0589  error=     9218.44
  470/500   0:00:32  loss=    -67.1698  error=     9199.84
  475/500   0:00:32  loss=    -71.1264  error=      9228.5
  480/500   0:00:32  loss=    -71.3954  error=     9199.72
  485/500   0:00:33  loss=    -72.1448  error=     9187.67
  490/500   0:00:33  loss=    -72.9153  error=     9159.87
  495/500   0:00:33  loss=    -73.6752  error=     9130.59
  500/500   0:00:34  loss=    -74.0321  error=     9058.68
Finished

Optimization finished in 34.325 seconds
‣ Iterations: 500
‣ Final loss: -74.0321
‣ Final error: 9058.68
==================================================

Trial 5 of 5

Starting optimization using Adam
‣ Model: MOSM
‣ Channels: 4
‣ Parameters: 64
‣ Training points: 384
‣ Initial loss: 179.591
‣ Initial error: 42232.3

Start Adam:
    0/500   0:00:00  loss=     179.591  error=     42232.3
    5/500   0:00:00  loss=     132.845  error=     22787.5
   10/500   0:00:00  loss=     111.424  error=     17147.8
   15/500   0:00:01  loss=     87.5926  error=     15683.5
   20/500   0:00:01  loss=     73.0244  error=     15624.6
   25/500   0:00:01  loss=     60.5564  error=     15688.6
   30/500   0:00:02  loss=     47.1089  error=     14559.1
   35/500   0:00:02  loss=     34.6772  error=     15123.8
   40/500   0:00:02  loss=     23.5324  error=     14541.2
   45/500   0:00:02  loss=     13.7541  error=     14131.7
   50/500   0:00:03  loss=     3.74656  error=       13836
   55/500   0:00:03  loss=    -5.56858  error=     13112.3
   60/500   0:00:03  loss=    -12.4858  error=     13159.4
   65/500   0:00:04  loss=     -19.745  error=     13257.1
   70/500   0:00:04  loss=    -26.1462  error=       13108
   75/500   0:00:05  loss=     -30.609  error=     13016.1
   80/500   0:00:05  loss=    -29.3854  error=     12761.8
   85/500   0:00:05  loss=    -33.6132  error=     12954.2
   90/500   0:00:06  loss=    -35.3574  error=     12685.9
   95/500   0:00:06  loss=    -39.2283  error=       12043
  100/500   0:00:06  loss=    -44.7104  error=     12652.7
  105/500   0:00:07  loss=    -44.2339  error=       12785
  110/500   0:00:07  loss=    -48.9544  error=       12224
  115/500   0:00:07  loss=    -44.9019  error=     12935.9
  120/500   0:00:08  loss=    -43.3744  error=     12578.4
  125/500   0:00:08  loss=    -24.4976  error=     11358.3
  130/500   0:00:08  loss=    -39.5739  error=     12574.7
  135/500   0:00:09  loss=    -37.9563  error=     12540.9
  140/500   0:00:09  loss=    -39.5827  error=     13248.8
  145/500   0:00:09  loss=    -47.8027  error=     11355.1
  150/500   0:00:10  loss=    -46.3567  error=     11143.9
  155/500   0:00:10  loss=    -49.2222  error=     11463.9
  160/500   0:00:10  loss=    -51.2808  error=     11107.2
  165/500   0:00:11  loss=    -52.8895  error=     11121.2
  170/500   0:00:11  loss=      -53.97  error=     11036.3
  175/500   0:00:11  loss=    -55.0873  error=     10920.8
  180/500   0:00:12  loss=      -56.55  error=     10860.4
  185/500   0:00:12  loss=    -58.4322  error=       10929
  190/500   0:00:12  loss=    -59.1914  error=     11218.4
  195/500   0:00:13  loss=    -60.7237  error=     10813.6
  200/500   0:00:13  loss=    -64.4055  error=     9423.48
  205/500   0:00:13  loss=    -66.4557  error=     8699.76
  210/500   0:00:14  loss=    -66.5761  error=     8677.48
  215/500   0:00:14  loss=    -67.2527  error=      8508.6
  220/500   0:00:14  loss=    -66.0746  error=     8333.23
  225/500   0:00:14  loss=    -67.0434  error=     8361.26
  230/500   0:00:15  loss=    -68.2914  error=     8582.67
  235/500   0:00:15  loss=    -68.4515  error=     8637.09
  240/500   0:00:15  loss=    -69.6161  error=     8409.19
  245/500   0:00:16  loss=    -69.2554  error=     8446.87
  250/500   0:00:16  loss=     -68.696  error=     8285.36
  255/500   0:00:16  loss=    -70.9447  error=     8503.18
  260/500   0:00:17  loss=     -67.436  error=      8340.8
  265/500   0:00:17  loss=    -68.1025  error=      8072.3
  270/500   0:00:17  loss=     -71.102  error=     8505.55
  275/500   0:00:18  loss=    -66.2061  error=     8724.41
  280/500   0:00:18  loss=    -63.6987  error=     8554.98
  285/500   0:00:18  loss=    -68.5953  error=     8225.19
  290/500   0:00:19  loss=    -70.8703  error=     8095.88
  295/500   0:00:19  loss=    -72.0128  error=     8159.36
  300/500   0:00:19  loss=    -70.6962  error=     8150.65
  305/500   0:00:20  loss=    -68.5389  error=     8284.65
  310/500   0:00:20  loss=    -71.2253  error=     8080.74
  315/500   0:00:20  loss=     -72.574  error=     8161.02
  320/500   0:00:21  loss=    -72.8505  error=     8281.29
  325/500   0:00:21  loss=    -72.8953  error=     8231.28
  330/500   0:00:21  loss=    -73.2328  error=     8203.88
  335/500   0:00:22  loss=    -73.2039  error=      8172.2
  340/500   0:00:22  loss=    -73.5638  error=     8155.98
  345/500   0:00:23  loss=    -73.5883  error=     8152.02
  350/500   0:00:23  loss=    -70.8336  error=     8271.36
  355/500   0:00:23  loss=    -72.6282  error=     8126.67
  360/500   0:00:24  loss=    -71.4298  error=     8207.49
  365/500   0:00:24  loss=    -70.4135  error=     8335.94
  370/500   0:00:24  loss=    -71.2351  error=     8272.03
  375/500   0:00:24  loss=    -72.7166  error=     8141.71
  380/500   0:00:25  loss=    -73.3215  error=     8235.36
  385/500   0:00:25  loss=     -72.853  error=     8315.49
  390/500   0:00:25  loss=    -72.7647  error=     8209.39
  395/500   0:00:26  loss=    -72.9522  error=     8206.18
  400/500   0:00:26  loss=    -73.4575  error=     8190.53
  405/500   0:00:26  loss=    -73.5715  error=     8205.82
  410/500   0:00:27  loss=    -73.5943  error=      8192.3
  415/500   0:00:27  loss=    -73.7785  error=     8183.56
  420/500   0:00:27  loss=    -73.8001  error=     8184.73
  425/500   0:00:28  loss=    -73.8128  error=     8188.82
  430/500   0:00:28  loss=    -73.8718  error=     8175.98
  435/500   0:00:28  loss=    -73.9106  error=      8176.3
  440/500   0:00:29  loss=    -73.9232  error=     8180.82
  445/500   0:00:29  loss=    -73.9352  error=     8179.39
  450/500   0:00:29  loss=    -73.9513  error=     8182.08
  455/500   0:00:30  loss=    -73.9626  error=     8184.29
  460/500   0:00:30  loss=    -73.9771  error=     8182.81
  465/500   0:00:30  loss=    -73.9918  error=     8183.97
  470/500   0:00:31  loss=    -74.0067  error=     8183.51
  475/500   0:00:31  loss=    -74.0226  error=     8182.77
  480/500   0:00:32  loss=    -74.0395  error=      8182.2
  485/500   0:00:32  loss=    -74.0548  error=     8181.28
  490/500   0:00:32  loss=    -74.0135  error=     8189.53
  495/500   0:00:33  loss=    -73.8313  error=      8182.4
  500/500   0:00:33  loss=    -74.0109  error=     8184.94
Finished

Optimization finished in 33.939 seconds
‣ Iterations: 500
‣ Final loss: -74.0109
‣ Final error: 8184.94
==================================================
In [5]:
pd.DataFrame(np.c_[mosm_mae.mean(1), mosm_rmse.mean(1), mosm_mape.mean(1),
             mosm_mae.std(1), mosm_rmse.std(1), mosm_mape.std(1)],
             columns=['MAE', 'RMSE', 'MAPE', 'MAE std', 'RMSE std', 'MAPE std'])
Out[5]:
MAE RMSE MAPE MAE std RMSE std MAPE std
0 46.314278 57.892007 3.113270 63.795370 78.749795 2.295002
1 50.125236 63.222404 3.066350 70.697668 88.796076 2.055261
2 37.255495 50.616210 3.092133 48.194712 66.700602 2.430008
3 41.798247 56.143231 3.150469 52.945970 70.728034 2.194666
4 39.563579 51.779989 3.081833 52.638941 68.639045 2.472799
In [6]:
best_mosm = mosm_models[np.argmin(mosm_mape.mean(1))]
best_mosm.plot_prediction();

Correlation of channels evaluated at zero¶

In [18]:
best_mosm.plot_correlation();

Cross spectral mixture kernel¶

In [8]:
csm_models = [None] * n_trials
csm_mae = np.zeros((n_trials,4))
csm_rmse = np.zeros((n_trials,4))
csm_mape = np.zeros((n_trials,4))

for n in range(n_trials):
    csm_dataset = dataset.copy()
    csm_dataset['Gold'].remove_randomly(pct=0.6)
    csm_dataset['Oil'].remove_randomly(pct=0.3)
    csm_dataset['NASDAQ'].remove_randomly(pct=0.6)
    csm_dataset['USD'].remove_randomly(pct=0.6)
    
    print('\nTrial', n+1, 'of', n_trials)
    csm = mogptk.CSM(csm_dataset, Q=Q)
    csm.init_parameters(init_method)
    try:
        csm.train(method=method, lr=lr, iters=iters, verbose=True, error='MSE')
    except mogptk.CholeskyException:
        csm_mape[n,:] = 100.0
        continue
    finally:
        print('=' * 50)
    
    error = mogptk.error(csm, per_channel=True)[0]
    csm_mae[n,:] = np.array([item['MAE'] for item in error])
    csm_rmse[n,:] = np.array([item['RMSE'] for item in error])
    csm_mape[n,:] = np.array([item['MAPE'] for item in error])
    csm_models[n] = csm
Trial 1 of 5

Starting optimization using Adam
‣ Model: CSM
‣ Channels: 4
‣ Parameters: 34
‣ Training points: 382
‣ Initial loss: 236.91
‣ Initial error:  36518

Start Adam:
    0/500   0:00:00  loss=      236.91  error=       36518
    5/500   0:00:00  loss=     128.508  error=     21722.5
   10/500   0:00:01  loss=     110.702  error=     26601.4
   15/500   0:00:01  loss=     89.7784  error=     21098.6
   20/500   0:00:02  loss=     74.0984  error=     18840.2
   25/500   0:00:03  loss=     60.6705  error=     17255.8
   30/500   0:00:03  loss=     46.8409  error=     17574.5
   35/500   0:00:04  loss=     32.5598  error=     16605.1
   40/500   0:00:05  loss=     20.7072  error=     16614.4
   45/500   0:00:05  loss=     10.3136  error=     14897.6
   50/500   0:00:06  loss=    0.472741  error=       14820
   55/500   0:00:06  loss=    -7.57304  error=     14389.7
   60/500   0:00:07  loss=    -14.2497  error=     13731.4
   65/500   0:00:08  loss=    -19.9383  error=     14039.7
   70/500   0:00:08  loss=    -24.3281  error=     13680.7
   75/500   0:00:09  loss=    -27.8145  error=     13872.4
   80/500   0:00:09  loss=    -30.4743  error=     13874.1
   85/500   0:00:10  loss=    -32.4746  error=       13962
   90/500   0:00:10  loss=    -33.9027  error=     14026.5
   95/500   0:00:11  loss=    -34.8994  error=     14080.3
  100/500   0:00:12  loss=    -35.5703  error=     14080.1
  105/500   0:00:12  loss=    -35.0256  error=     14582.9
  110/500   0:00:13  loss=    -35.6108  error=     14522.2
  115/500   0:00:13  loss=    -35.4707  error=     14514.3
  120/500   0:00:14  loss=    -36.6482  error=     14332.3
  125/500   0:00:15  loss=    -36.6023  error=     14269.6
  130/500   0:00:15  loss=    -36.8103  error=     14331.2
  135/500   0:00:16  loss=      -36.99  error=     14455.8
  140/500   0:00:16  loss=    -37.0611  error=     14515.8
  145/500   0:00:17  loss=    -37.1602  error=     14514.6
  150/500   0:00:18  loss=    -37.2562  error=     14524.1
  155/500   0:00:18  loss=    -37.3323  error=     14524.1
  160/500   0:00:19  loss=    -37.3973  error=     14538.9
  165/500   0:00:19  loss=    -37.4645  error=       14562
  170/500   0:00:20  loss=    -37.5283  error=     14556.6
  175/500   0:00:21  loss=    -37.5871  error=     14547.5
  180/500   0:00:21  loss=    -37.6416  error=     14548.9
  185/500   0:00:22  loss=    -37.6915  error=     14566.9
  190/500   0:00:22  loss=    -37.7056  error=     14520.6
  195/500   0:00:23  loss=    -37.2937  error=       14696
  200/500   0:00:24  loss=    -37.5844  error=     14446.2
  205/500   0:00:24  loss=     -37.706  error=     14465.5
  210/500   0:00:25  loss=    -37.8997  error=     14555.3
  215/500   0:00:25  loss=    -37.8935  error=     14597.5
  220/500   0:00:26  loss=    -37.9819  error=     14476.3
  225/500   0:00:27  loss=    -38.0702  error=     14458.2
  230/500   0:00:27  loss=    -38.0276  error=     14548.9
  235/500   0:00:28  loss=    -37.8184  error=     14389.3
  240/500   0:00:28  loss=    -37.2707  error=     14734.2
  245/500   0:00:29  loss=    -37.9319  error=     14563.3
  250/500   0:00:30  loss=    -38.2894  error=     14417.3
  255/500   0:00:30  loss=    -38.1518  error=     14432.5
  260/500   0:00:31  loss=    -37.9571  error=     14330.1
  265/500   0:00:31  loss=    -38.1024  error=     14447.2
  270/500   0:00:32  loss=     -38.346  error=     14457.9
  275/500   0:00:33  loss=    -38.4373  error=       14403
  280/500   0:00:33  loss=    -38.3851  error=     14360.5
  285/500   0:00:34  loss=     -38.222  error=     14339.6
  290/500   0:00:34  loss=    -38.3323  error=       14476
  295/500   0:00:35  loss=    -38.4447  error=     14373.9
  300/500   0:00:35  loss=    -38.6386  error=     14342.8
  305/500   0:00:36  loss=     -38.613  error=     14352.5
  310/500   0:00:37  loss=    -38.6899  error=     14300.2
  315/500   0:00:37  loss=    -38.2937  error=     14509.7
  320/500   0:00:38  loss=    -37.8954  error=     14309.1
  325/500   0:00:38  loss=     -38.457  error=     14342.8
  330/500   0:00:39  loss=    -38.3734  error=       14420
  335/500   0:00:40  loss=    -38.5722  error=     14376.8
  340/500   0:00:40  loss=     -38.893  error=     14326.6
  345/500   0:00:41  loss=    -38.9263  error=       14298
  350/500   0:00:41  loss=     -38.904  error=     14301.3
  355/500   0:00:42  loss=    -38.8739  error=     14329.8
  360/500   0:00:43  loss=    -38.3827  error=     14292.8
  365/500   0:00:43  loss=    -38.8968  error=     14307.5
  370/500   0:00:44  loss=     -38.922  error=     14370.5
  375/500   0:00:44  loss=     -39.017  error=     14310.2
  380/500   0:00:45  loss=    -39.0757  error=     14292.8
  385/500   0:00:46  loss=    -39.0938  error=     14333.2
  390/500   0:00:46  loss=    -39.1424  error=       14262
  395/500   0:00:47  loss=    -39.0271  error=     14380.9
  400/500   0:00:47  loss=    -38.1357  error=     14238.2
  405/500   0:00:48  loss=    -38.9835  error=     14317.5
  410/500   0:00:49  loss=     -38.426  error=     14548.5
  415/500   0:00:49  loss=    -38.6621  error=     14337.2
  420/500   0:00:50  loss=    -39.2349  error=     14314.4
  425/500   0:00:51  loss=    -39.2101  error=     14350.4
  430/500   0:00:51  loss=    -39.2124  error=     14251.7
  435/500   0:00:52  loss=    -39.2172  error=     14196.7
  440/500   0:00:52  loss=    -39.3037  error=     14290.4
  445/500   0:00:53  loss=    -39.1644  error=     14324.6
  450/500   0:00:54  loss=    -39.0529  error=       14399
  455/500   0:00:54  loss=    -39.3849  error=     14250.4
  460/500   0:00:55  loss=    -39.4384  error=     14269.2
  465/500   0:00:55  loss=    -39.3679  error=     14302.8
  470/500   0:00:56  loss=    -39.2368  error=     14260.5
  475/500   0:00:57  loss=    -39.2318  error=     14355.7
  480/500   0:00:57  loss=    -39.3719  error=     14230.3
  485/500   0:00:58  loss=    -39.3764  error=       14320
  490/500   0:00:59  loss=    -39.2455  error=     14351.1
  495/500   0:00:59  loss=    -39.2794  error=     14254.9
  500/500   0:01:00  loss=    -39.5137  error=     14311.1
Finished

Optimization finished in 1 minute
‣ Iterations: 500
‣ Final loss: -39.5137
‣ Final error: 14311.1
==================================================

Trial 2 of 5

Starting optimization using Adam
‣ Model: CSM
‣ Channels: 4
‣ Parameters: 34
‣ Training points: 383
‣ Initial loss: 224.772
‣ Initial error: 44138.8

Start Adam:
    0/500   0:00:00  loss=     224.772  error=     44138.8
    5/500   0:00:00  loss=     110.697  error=     31970.8
   10/500   0:00:01  loss=     102.192  error=     22251.7
   15/500   0:00:01  loss=     84.7974  error=     20791.9
   20/500   0:00:02  loss=     68.2204  error=       19835
   25/500   0:00:03  loss=     53.4965  error=     20761.9
   30/500   0:00:03  loss=     40.1748  error=     18670.1
   35/500   0:00:04  loss=     26.6689  error=       17645
   40/500   0:00:04  loss=     15.2592  error=     17195.5
   45/500   0:00:05  loss=     4.06056  error=     16499.9
   50/500   0:00:06  loss=    -6.43573  error=     15628.1
   55/500   0:00:06  loss=    -15.4612  error=     15332.7
   60/500   0:00:07  loss=    -23.2257  error=     15179.2
   65/500   0:00:07  loss=     -28.659  error=     15354.5
   70/500   0:00:08  loss=    -34.8693  error=     15491.7
   75/500   0:00:09  loss=    -39.8635  error=     15468.9
   80/500   0:00:09  loss=    -43.4949  error=       15559
   85/500   0:00:10  loss=    -46.3278  error=     15716.9
   90/500   0:00:10  loss=    -48.5352  error=     15875.3
   95/500   0:00:11  loss=    -50.4677  error=     16002.7
  100/500   0:00:11  loss=    -51.8643  error=     16133.6
  105/500   0:00:12  loss=    -53.0393  error=     16267.6
  110/500   0:00:13  loss=    -53.9615  error=     16392.5
  115/500   0:00:13  loss=    -53.8963  error=     16506.6
  120/500   0:00:14  loss=    -55.0965  error=     16578.4
  125/500   0:00:14  loss=    -55.5457  error=     16697.5
  130/500   0:00:15  loss=    -56.0323  error=     16784.1
  135/500   0:00:16  loss=    -56.3531  error=     16861.9
  140/500   0:00:16  loss=    -56.7054  error=     16930.1
  145/500   0:00:17  loss=    -56.8206  error=     17008.5
  150/500   0:00:18  loss=    -56.9799  error=       17059
  155/500   0:00:18  loss=    -57.1258  error=     17132.5
  160/500   0:00:19  loss=    -57.1046  error=     17185.5
  165/500   0:00:19  loss=      -56.77  error=     17304.1
  170/500   0:00:20  loss=    -57.5113  error=     17276.9
  175/500   0:00:21  loss=    -57.3512  error=     17357.7
  180/500   0:00:22  loss=    -57.6016  error=     17394.6
  185/500   0:00:22  loss=    -57.6352  error=     17450.1
  190/500   0:00:23  loss=    -57.8171  error=     17500.5
  195/500   0:00:23  loss=    -57.7659  error=     17571.1
  200/500   0:00:24  loss=    -57.6175  error=     17642.4
  205/500   0:00:25  loss=    -57.7442  error=     17705.2
  210/500   0:00:25  loss=    -58.1063  error=     17741.6
  215/500   0:00:26  loss=    -57.9877  error=     17815.3
  220/500   0:00:26  loss=    -57.9163  error=     17906.2
  225/500   0:00:27  loss=    -58.2029  error=     17942.3
  230/500   0:00:28  loss=    -58.3835  error=     18011.7
  235/500   0:00:28  loss=    -58.3515  error=     18093.6
  240/500   0:00:29  loss=    -57.7696  error=     18190.9
  245/500   0:00:29  loss=    -57.8384  error=     18296.4
  250/500   0:00:30  loss=    -57.8811  error=     18336.8
  255/500   0:00:31  loss=    -58.5183  error=       18399
  260/500   0:00:31  loss=    -58.7362  error=     18442.1
  265/500   0:00:32  loss=    -58.6516  error=     18523.7
  270/500   0:00:33  loss=     -58.788  error=     18608.4
  275/500   0:00:33  loss=    -58.6218  error=     18662.6
  280/500   0:00:34  loss=    -58.3804  error=     18796.5
  285/500   0:00:34  loss=    -58.7535  error=     18843.5
  290/500   0:00:35  loss=    -59.0676  error=     18905.4
  295/500   0:00:36  loss=    -59.0944  error=     18970.4
  300/500   0:00:36  loss=    -58.7932  error=     19045.4
  305/500   0:00:37  loss=    -57.8606  error=     19203.3
  310/500   0:00:37  loss=    -59.2688  error=     19184.2
  315/500   0:00:38  loss=    -59.1166  error=       19244
  320/500   0:00:39  loss=    -59.1521  error=     19328.3
  325/500   0:00:39  loss=    -59.3517  error=     19399.3
  330/500   0:00:40  loss=    -59.4784  error=     19470.9
  335/500   0:00:40  loss=     -59.443  error=     19537.5
  340/500   0:00:41  loss=    -59.5871  error=     19601.6
  345/500   0:00:42  loss=    -59.5807  error=     19658.1
  350/500   0:00:42  loss=    -59.5827  error=     19750.9
  355/500   0:00:43  loss=     -58.735  error=     19780.2
  360/500   0:00:43  loss=    -57.9138  error=     20028.4
  365/500   0:00:44  loss=    -58.3166  error=     20015.3
  370/500   0:00:45  loss=    -59.0997  error=     20041.4
  375/500   0:00:45  loss=    -59.5427  error=     20039.4
  380/500   0:00:46  loss=    -59.7666  error=     20080.2
  385/500   0:00:46  loss=    -59.8466  error=     20101.9
  390/500   0:00:47  loss=    -59.8722  error=     20152.2
  395/500   0:00:48  loss=    -59.9179  error=     20205.9
  400/500   0:00:48  loss=    -59.9372  error=     20275.3
  405/500   0:00:49  loss=    -59.9713  error=     20310.5
  410/500   0:00:50  loss=    -60.0152  error=     20313.9
  415/500   0:00:50  loss=    -60.0326  error=     20342.2
  420/500   0:00:51  loss=    -60.0584  error=     20381.1
  425/500   0:00:51  loss=    -60.0619  error=     20393.6
  430/500   0:00:52  loss=     -59.955  error=     20472.3
  435/500   0:00:52  loss=    -58.5506  error=     20348.1
  440/500   0:00:53  loss=    -59.4679  error=     20545.5
  445/500   0:00:54  loss=    -59.3133  error=       20630
  450/500   0:00:54  loss=    -59.5439  error=     20633.6
  455/500   0:00:55  loss=    -59.9341  error=     20569.5
  460/500   0:00:55  loss=    -60.1606  error=       20506
  465/500   0:00:56  loss=     -60.215  error=     20460.5
  470/500   0:00:57  loss=    -60.1787  error=     20448.8
  475/500   0:00:57  loss=    -60.2064  error=     20448.4
  480/500   0:00:58  loss=    -60.2724  error=     20476.5
  485/500   0:00:59  loss=    -60.2779  error=     20514.5
  490/500   0:00:59  loss=    -60.2779  error=     20511.1
  495/500   0:01:00  loss=     -60.144  error=     20511.1
  500/500   0:01:00  loss=     -59.204  error=     20488.6
Finished

Optimization finished in 1 minute
‣ Iterations: 500
‣ Final loss: -59.204
‣ Final error: 20488.6
==================================================

Trial 3 of 5

Starting optimization using Adam
‣ Model: CSM
‣ Channels: 4
‣ Parameters: 34
‣ Training points: 383
‣ Initial loss: 221.489
‣ Initial error: 34086.6

Start Adam:
    0/500   0:00:00  loss=     221.489  error=     34086.6
    5/500   0:00:00  loss=     126.189  error=     21413.8
   10/500   0:00:01  loss=     97.9359  error=     20677.6
   15/500   0:00:01  loss=     75.3474  error=     19608.2
   20/500   0:00:02  loss=     60.9586  error=     19315.4
   25/500   0:00:03  loss=     49.3987  error=     21026.6
   30/500   0:00:03  loss=     36.9016  error=     17367.3
   35/500   0:00:04  loss=     25.6153  error=     17996.4
   40/500   0:00:05  loss=     15.8948  error=     16887.4
   45/500   0:00:05  loss=      7.3212  error=     16963.2
   50/500   0:00:06  loss=   -0.347473  error=     16484.3
   55/500   0:00:07  loss=    -7.02624  error=     15132.8
   60/500   0:00:07  loss=    -13.3846  error=       13421
   65/500   0:00:08  loss=    -18.5297  error=     11549.6
   70/500   0:00:08  loss=    -22.8229  error=     12418.5
   75/500   0:00:09  loss=    -25.8794  error=     12444.5
   80/500   0:00:10  loss=    -27.8574  error=     12224.8
   85/500   0:00:10  loss=    -26.5457  error=     12420.1
   90/500   0:00:11  loss=    -29.5122  error=       12559
   95/500   0:00:11  loss=    -30.2869  error=       12925
  100/500   0:00:12  loss=    -31.8311  error=     12886.7
  105/500   0:00:13  loss=    -32.2057  error=     12853.9
  110/500   0:00:13  loss=    -32.8214  error=     13200.1
  115/500   0:00:14  loss=    -33.5851  error=     13333.3
  120/500   0:00:15  loss=    -34.3249  error=     13675.8
  125/500   0:00:15  loss=    -35.7691  error=     14584.5
  130/500   0:00:16  loss=    -44.1938  error=     15870.3
  135/500   0:00:17  loss=    -45.3824  error=     17375.8
  140/500   0:00:17  loss=     -48.185  error=     16352.7
  145/500   0:00:18  loss=    -49.3444  error=     16187.9
  150/500   0:00:18  loss=    -49.3205  error=     17081.9
  155/500   0:00:19  loss=    -49.2945  error=     16279.3
  160/500   0:00:20  loss=    -51.0072  error=     16620.9
  165/500   0:00:20  loss=    -50.9456  error=     16595.6
  170/500   0:00:21  loss=    -50.9065  error=     16398.8
  175/500   0:00:21  loss=    -51.2649  error=     16447.5
  180/500   0:00:22  loss=    -51.5349  error=     16593.1
  185/500   0:00:23  loss=    -51.6802  error=       16496
  190/500   0:00:23  loss=    -51.7602  error=     16494.6
  195/500   0:00:24  loss=    -51.8098  error=     16569.5
  200/500   0:00:24  loss=    -51.8628  error=     16473.4
  205/500   0:00:25  loss=    -51.9351  error=     16506.5
  210/500   0:00:26  loss=    -52.0212  error=     16543.6
  215/500   0:00:26  loss=     -52.073  error=     16531.9
  220/500   0:00:27  loss=     -52.152  error=     16540.3
  225/500   0:00:28  loss=    -52.2124  error=     16527.4
  230/500   0:00:28  loss=    -52.2629  error=     16544.5
  235/500   0:00:29  loss=    -52.1525  error=     16490.9
  240/500   0:00:29  loss=    -51.0405  error=     16706.6
  245/500   0:00:30  loss=    -52.5304  error=     16519.4
  250/500   0:00:31  loss=    -52.4257  error=       16455
  255/500   0:00:31  loss=    -52.4153  error=       16456
  260/500   0:00:32  loss=     -52.714  error=     16475.6
  265/500   0:00:32  loss=    -52.9848  error=     16510.9
  270/500   0:00:33  loss=    -53.0544  error=     16547.4
  275/500   0:00:34  loss=    -53.2457  error=     16544.6
  280/500   0:00:34  loss=    -53.4271  error=     16508.3
  285/500   0:00:35  loss=    -53.6243  error=     16481.9
  290/500   0:00:35  loss=    -53.8433  error=     16552.2
  295/500   0:00:36  loss=    -54.0926  error=     16529.2
  300/500   0:00:37  loss=    -54.3499  error=     16529.1
  305/500   0:00:37  loss=    -54.5288  error=     16634.6
  310/500   0:00:38  loss=    -52.6818  error=     16377.6
  315/500   0:00:38  loss=    -53.8535  error=     16281.3
  320/500   0:00:39  loss=     -54.909  error=     16315.2
  325/500   0:00:40  loss=    -55.4521  error=     16349.9
  330/500   0:00:40  loss=    -56.1535  error=     16256.9
  335/500   0:00:41  loss=    -56.8734  error=       16050
  340/500   0:00:41  loss=    -57.5051  error=     15876.1
  345/500   0:00:42  loss=    -58.0073  error=     15800.1
  350/500   0:00:43  loss=    -58.1308  error=       15682
  355/500   0:00:43  loss=    -57.1941  error=     15848.6
  360/500   0:00:44  loss=    -59.2064  error=     15670.5
  365/500   0:00:44  loss=    -59.3731  error=     15641.8
  370/500   0:00:45  loss=    -59.4356  error=     15687.3
  375/500   0:00:46  loss=    -60.0644  error=     15710.3
  380/500   0:00:46  loss=    -60.1338  error=     15780.3
  385/500   0:00:47  loss=    -60.4319  error=     15808.2
  390/500   0:00:47  loss=    -60.5021  error=     15823.2
  395/500   0:00:48  loss=    -60.7494  error=     15888.1
  400/500   0:00:49  loss=     -60.941  error=       15899
  405/500   0:00:49  loss=    -61.0718  error=     15945.2
  410/500   0:00:50  loss=    -60.8483  error=     15960.6
  415/500   0:00:50  loss=    -58.4882  error=     16213.4
  420/500   0:00:51  loss=    -60.7953  error=       16079
  425/500   0:00:52  loss=    -61.5894  error=     16045.6
  430/500   0:00:52  loss=    -61.7242  error=       16068
  435/500   0:00:53  loss=    -61.7194  error=     16103.6
  440/500   0:00:53  loss=    -61.8189  error=     16092.4
  445/500   0:00:54  loss=    -62.1186  error=     16139.6
  450/500   0:00:55  loss=    -62.3853  error=     16187.7
  455/500   0:00:55  loss=    -62.5526  error=     16242.4
  460/500   0:00:56  loss=    -62.9368  error=     16211.9
  465/500   0:00:56  loss=    -63.3464  error=     16156.3
  470/500   0:00:57  loss=    -63.7021  error=     16136.4
  475/500   0:00:58  loss=    -63.7333  error=     16170.9
  480/500   0:00:58  loss=    -62.2974  error=     16109.1
  485/500   0:00:59  loss=    -64.4789  error=     16181.4
  490/500   0:00:59  loss=    -64.0383  error=     16308.3
  495/500   0:01:00  loss=    -64.3097  error=     16290.4
  500/500   0:01:01  loss=    -64.7846  error=       16240
Finished

Optimization finished in 1 minute 1 second
‣ Iterations: 500
‣ Final loss: -64.7846
‣ Final error:  16240
==================================================

Trial 4 of 5

Starting optimization using Adam
‣ Model: CSM
‣ Channels: 4
‣ Parameters: 34
‣ Training points: 377
‣ Initial loss: 205.333
‣ Initial error: 38513.5

Start Adam:
    0/500   0:00:00  loss=     205.333  error=     38513.5
    5/500   0:00:00  loss=     128.642  error=     27013.3
   10/500   0:00:01  loss=     97.3435  error=     20576.1
   15/500   0:00:01  loss=     84.6528  error=     29693.7
   20/500   0:00:02  loss=     68.6479  error=     23766.4
   25/500   0:00:03  loss=     52.5288  error=     23462.7
   30/500   0:00:03  loss=     39.0599  error=     23350.6
   35/500   0:00:04  loss=     25.9803  error=     20921.6
   40/500   0:00:05  loss=     14.2643  error=     21219.9
   45/500   0:00:05  loss=     3.88135  error=     19298.6
   50/500   0:00:06  loss=    -5.25765  error=     19456.9
   55/500   0:00:07  loss=    -13.3754  error=     18680.6
   60/500   0:00:07  loss=    -20.4335  error=     18362.3
   65/500   0:00:08  loss=    -26.4261  error=     18020.4
   70/500   0:00:08  loss=    -31.4184  error=     17954.3
   75/500   0:00:09  loss=    -35.5735  error=       17625
   80/500   0:00:10  loss=    -38.8749  error=     17442.3
   85/500   0:00:10  loss=    -41.8302  error=     17577.6
   90/500   0:00:11  loss=    -44.1373  error=     17561.6
   95/500   0:00:11  loss=    -46.0096  error=     17517.9
  100/500   0:00:12  loss=    -47.5438  error=     17520.6
  105/500   0:00:13  loss=    -48.8129  error=     17690.5
  110/500   0:00:13  loss=    -49.8388  error=       17794
  115/500   0:00:14  loss=    -50.7894  error=     17760.9
  120/500   0:00:15  loss=    -51.4671  error=     17653.8
  125/500   0:00:15  loss=    -51.9399  error=     18143.7
  130/500   0:00:16  loss=    -52.8404  error=     17825.2
  135/500   0:00:16  loss=    -53.1856  error=     17760.8
  140/500   0:00:17  loss=    -52.7705  error=     18434.6
  145/500   0:00:18  loss=    -54.0091  error=       18104
  150/500   0:00:18  loss=    -54.3976  error=     17896.8
  155/500   0:00:19  loss=    -54.5679  error=     17844.7
  160/500   0:00:19  loss=    -54.8649  error=     17926.7
  165/500   0:00:20  loss=    -55.2144  error=     17950.8
  170/500   0:00:21  loss=    -55.5271  error=     17916.4
  175/500   0:00:21  loss=    -55.3509  error=       18279
  180/500   0:00:22  loss=    -55.6627  error=     17538.1
  185/500   0:00:22  loss=    -56.1439  error=     16901.3
  190/500   0:00:23  loss=    -56.7079  error=     16812.2
  195/500   0:00:24  loss=    -57.2436  error=     17184.6
  200/500   0:00:24  loss=     -57.737  error=     16926.6
  205/500   0:00:25  loss=    -58.1267  error=     17014.9
  210/500   0:00:25  loss=    -58.4025  error=     17161.4
  215/500   0:00:26  loss=    -58.5968  error=     17060.2
  220/500   0:00:26  loss=    -58.8326  error=       17004
  225/500   0:00:27  loss=    -58.9803  error=       16781
  230/500   0:00:28  loss=    -59.0872  error=     17082.8
  235/500   0:00:28  loss=    -58.9429  error=     16531.2
  240/500   0:00:29  loss=    -58.7311  error=     17438.5
  245/500   0:00:29  loss=    -59.4671  error=     17046.8
  250/500   0:00:30  loss=    -59.4387  error=     16674.8
  255/500   0:00:31  loss=    -59.7153  error=     16696.1
  260/500   0:00:31  loss=    -59.7652  error=     17042.9
  265/500   0:00:32  loss=    -59.9956  error=     16684.4
  270/500   0:00:32  loss=    -60.0897  error=     16661.9
  275/500   0:00:33  loss=    -60.0593  error=       16980
  280/500   0:00:34  loss=    -58.8829  error=     16251.3
  285/500   0:00:34  loss=    -60.3408  error=     16824.6
  290/500   0:00:35  loss=    -59.8825  error=     17234.5
  295/500   0:00:35  loss=    -60.0371  error=     17143.3
  300/500   0:00:36  loss=    -60.4928  error=     16988.1
  305/500   0:00:36  loss=    -60.6204  error=     16753.4
  310/500   0:00:37  loss=    -60.5673  error=     16590.7
  315/500   0:00:38  loss=    -60.8261  error=     16712.2
  320/500   0:00:38  loss=    -60.8191  error=       16940
  325/500   0:00:39  loss=    -60.9883  error=     16678.4
  330/500   0:00:39  loss=    -61.0288  error=     16613.3
  335/500   0:00:40  loss=    -60.9184  error=     16996.4
  340/500   0:00:41  loss=    -60.0337  error=     16373.5
  345/500   0:00:41  loss=    -61.1649  error=     16926.9
  350/500   0:00:42  loss=    -60.9045  error=     17123.9
  355/500   0:00:42  loss=    -61.1752  error=     16686.6
  360/500   0:00:43  loss=    -61.3068  error=     16575.2
  365/500   0:00:43  loss=    -61.3856  error=     16838.4
  370/500   0:00:44  loss=    -61.4544  error=     16912.1
  375/500   0:00:45  loss=    -61.4689  error=     16551.2
  380/500   0:00:45  loss=    -61.4368  error=     16938.4
  385/500   0:00:46  loss=    -60.5754  error=       16410
  390/500   0:00:46  loss=    -61.4482  error=     17142.7
  395/500   0:00:47  loss=    -61.3227  error=     16930.9
  400/500   0:00:48  loss=     -61.538  error=     16569.8
  405/500   0:00:48  loss=    -61.7973  error=     16701.1
  410/500   0:00:49  loss=    -61.7616  error=     16875.9
  415/500   0:00:49  loss=    -61.8695  error=     16712.7
  420/500   0:00:50  loss=    -61.9259  error=     16684.8
  425/500   0:00:51  loss=    -61.9791  error=     16718.9
  430/500   0:00:51  loss=    -62.0234  error=     16770.3
  435/500   0:00:52  loss=     -61.997  error=     16547.4
  440/500   0:00:52  loss=    -60.5985  error=     17574.2
  445/500   0:00:53  loss=    -61.2473  error=     16618.5
  450/500   0:00:54  loss=    -61.1043  error=     16426.3
  455/500   0:00:54  loss=    -61.9454  error=     16371.8
  460/500   0:00:55  loss=    -62.1436  error=     16886.9
  465/500   0:00:55  loss=    -62.0837  error=     16863.7
  470/500   0:00:56  loss=    -62.0671  error=     16892.1
  475/500   0:00:57  loss=    -62.1402  error=     16977.3
  480/500   0:00:57  loss=     -62.224  error=     16812.6
  485/500   0:00:58  loss=    -62.2914  error=     16776.4
  490/500   0:00:59  loss=    -62.3523  error=     16773.8
  495/500   0:00:59  loss=    -62.3959  error=     16738.8
  500/500   0:01:00  loss=     -62.434  error=     16703.1
Finished

Optimization finished in 1 minute
‣ Iterations: 500
‣ Final loss: -62.434
‣ Final error: 16703.1
==================================================

Trial 5 of 5

Starting optimization using Adam
‣ Model: CSM
‣ Channels: 4
‣ Parameters: 34
‣ Training points: 383
‣ Initial loss: 210.353
‣ Initial error: 35497.1

Start Adam:
    0/500   0:00:00  loss=     210.353  error=     35497.1
    5/500   0:00:00  loss=     113.733  error=     14620.9
   10/500   0:00:01  loss=     89.5163  error=     21762.7
   15/500   0:00:01  loss=     69.9942  error=     15618.9
   20/500   0:00:02  loss=     53.9041  error=       16227
   25/500   0:00:03  loss=     39.9231  error=     15636.3
   30/500   0:00:03  loss=     27.8082  error=     16020.9
   35/500   0:00:04  loss=     16.4591  error=     15477.3
   40/500   0:00:05  loss=     6.26384  error=     15864.5
   45/500   0:00:05  loss=    -2.99196  error=     15936.4
   50/500   0:00:06  loss=     -10.345  error=     16246.3
   55/500   0:00:06  loss=    -16.7665  error=       16402
   60/500   0:00:07  loss=    -22.0857  error=     16610.6
   65/500   0:00:08  loss=    -26.4409  error=     16610.9
   70/500   0:00:08  loss=    -31.6821  error=     16332.3
   75/500   0:00:09  loss=    -38.1326  error=     14922.4
   80/500   0:00:09  loss=    -40.9992  error=       13493
   85/500   0:00:10  loss=    -43.9309  error=     13252.9
   90/500   0:00:11  loss=    -46.2388  error=     14622.1
   95/500   0:00:11  loss=    -48.2428  error=     13855.7
  100/500   0:00:12  loss=    -50.1655  error=     13323.4
  105/500   0:00:13  loss=    -51.3647  error=     13764.8
  110/500   0:00:13  loss=    -52.3036  error=     13332.4
  115/500   0:00:14  loss=    -51.4872  error=     13556.9
  120/500   0:00:14  loss=    -53.3076  error=     13383.6
  125/500   0:00:15  loss=    -54.2404  error=     13222.9
  130/500   0:00:16  loss=    -54.7046  error=     13157.6
  135/500   0:00:16  loss=    -54.9705  error=     13057.7
  140/500   0:00:17  loss=    -55.1404  error=       12887
  145/500   0:00:18  loss=    -55.1338  error=       12816
  150/500   0:00:18  loss=    -54.9677  error=     12743.6
  155/500   0:00:19  loss=    -55.3168  error=     12760.2
  160/500   0:00:19  loss=    -55.4344  error=     12867.7
  165/500   0:00:20  loss=    -55.5482  error=     12875.3
  170/500   0:00:21  loss=    -55.5033  error=     12784.7
  175/500   0:00:21  loss=    -55.7638  error=       12841
  180/500   0:00:22  loss=    -55.8995  error=     12799.8
  185/500   0:00:22  loss=    -55.9416  error=     12884.1
  190/500   0:00:23  loss=    -54.8984  error=     12820.8
  195/500   0:00:24  loss=    -56.1267  error=     12925.2
  200/500   0:00:24  loss=    -55.5643  error=     13163.5
  205/500   0:00:25  loss=    -56.0407  error=       12912
  210/500   0:00:25  loss=      -56.28  error=       12928
  215/500   0:00:26  loss=    -56.4111  error=     12840.6
  220/500   0:00:27  loss=    -56.5101  error=     12864.5
  225/500   0:00:27  loss=    -56.5623  error=     12890.1
  230/500   0:00:28  loss=    -56.6006  error=     12888.7
  235/500   0:00:28  loss=     -56.676  error=     12902.6
  240/500   0:00:29  loss=    -56.8014  error=     12867.5
  245/500   0:00:30  loss=    -56.8648  error=     12851.6
  250/500   0:00:30  loss=     -56.934  error=     12851.4
  255/500   0:00:31  loss=    -56.9831  error=     12884.4
  260/500   0:00:31  loss=    -57.0381  error=     12869.5
  265/500   0:00:32  loss=    -56.9991  error=       12918
  270/500   0:00:33  loss=    -56.2453  error=     12884.8
  275/500   0:00:33  loss=    -56.9691  error=     13024.7
  280/500   0:00:34  loss=    -56.6568  error=     13065.1
  285/500   0:00:34  loss=    -57.3369  error=     12882.2
  290/500   0:00:35  loss=    -56.8497  error=     12905.7
  295/500   0:00:36  loss=    -56.9087  error=     12742.6
  300/500   0:00:36  loss=    -57.3611  error=     12840.2
  305/500   0:00:37  loss=    -57.1889  error=     13017.4
  310/500   0:00:37  loss=    -57.4921  error=     12941.9
  315/500   0:00:38  loss=    -57.5732  error=     13002.6
  320/500   0:00:39  loss=    -57.5288  error=     12990.3
  325/500   0:00:39  loss=    -57.7513  error=     12952.7
  330/500   0:00:40  loss=    -57.8095  error=     12965.4
  335/500   0:00:41  loss=      -57.79  error=     12855.9
  340/500   0:00:41  loss=    -57.8571  error=     12956.8
  345/500   0:00:42  loss=    -57.7206  error=     12935.7
  350/500   0:00:42  loss=    -57.2288  error=     13160.9
  355/500   0:00:43  loss=    -57.8705  error=     12930.6
  360/500   0:00:44  loss=    -57.8862  error=       12908
  365/500   0:00:44  loss=      -57.82  error=     12970.3
  370/500   0:00:45  loss=    -58.1626  error=     12990.5
  375/500   0:00:46  loss=    -58.1851  error=     12970.4
  380/500   0:00:46  loss=    -58.0919  error=     13001.7
  385/500   0:00:47  loss=     -57.571  error=     12876.4
  390/500   0:00:47  loss=    -58.0711  error=     13045.8
  395/500   0:00:48  loss=    -58.1693  error=     13153.3
  400/500   0:00:49  loss=    -58.2676  error=     12948.7
  405/500   0:00:49  loss=    -58.1311  error=       12929
  410/500   0:00:50  loss=    -58.4891  error=     12920.7
  415/500   0:00:51  loss=     -58.344  error=     13127.7
  420/500   0:00:51  loss=    -58.3922  error=     12892.3
  425/500   0:00:52  loss=    -57.8964  error=     13105.2
  430/500   0:00:52  loss=    -58.0065  error=     12967.9
  435/500   0:00:53  loss=    -58.5978  error=       13051
  440/500   0:00:54  loss=    -58.4898  error=     13098.8
  445/500   0:00:54  loss=    -58.4316  error=     12863.3
  450/500   0:00:55  loss=    -58.7914  error=       13039
  455/500   0:00:56  loss=    -58.7789  error=       13072
  460/500   0:00:56  loss=    -58.6869  error=       12895
  465/500   0:00:57  loss=    -58.5275  error=     13181.5
  470/500   0:00:57  loss=      -58.34  error=     12977.5
  475/500   0:00:58  loss=    -57.8351  error=     12999.7
  480/500   0:00:59  loss=    -58.9658  error=     13007.2
  485/500   0:00:59  loss=    -58.6577  error=     13031.9
  490/500   0:01:00  loss=    -58.7719  error=     13115.4
  495/500   0:01:01  loss=    -58.8594  error=     12898.1
  500/500   0:01:01  loss=    -58.6563  error=     13131.1
Finished

Optimization finished in 1 minute 1 second
‣ Iterations: 500
‣ Final loss: -58.6563
‣ Final error: 13131.1
==================================================
In [9]:
pd.DataFrame(np.c_[csm_mae.mean(1), csm_rmse.mean(1), csm_mape.mean(1),
             csm_mae.std(1), csm_rmse.std(1), csm_mape.std(1)],
             columns=['MAE', 'RMSE', 'MAPE', 'MAE std', 'RMSE std', 'MAPE std'])
Out[9]:
MAE RMSE MAPE MAE std RMSE std MAPE std
0 56.033990 67.154186 3.473000 76.696496 92.272892 2.297696
1 61.847129 78.700941 3.365134 87.789977 111.322088 2.106615
2 57.728856 70.997269 3.352222 80.756686 98.550258 2.022270
3 53.498076 71.155602 3.595584 75.937080 101.457619 2.885670
4 53.759670 66.363301 3.850873 70.024947 86.537374 2.749064
In [10]:
best_csm = csm_models[np.argmin(csm_mape.mean(1))]
best_csm.plot_prediction();

Linear model of coregionalization with spectral mixture kernels¶

In [11]:
smlmc_models = [None] * n_trials
smlmc_mae = np.zeros((n_trials,4))
smlmc_rmse = np.zeros((n_trials,4))
smlmc_mape = np.zeros((n_trials,4))

for n in range(n_trials):
    smlmc_dataset = dataset.copy()
    smlmc_dataset['Gold'].remove_randomly(pct=0.6)
    smlmc_dataset['Oil'].remove_randomly(pct=0.3)
    smlmc_dataset['NASDAQ'].remove_randomly(pct=0.6)
    smlmc_dataset['USD'].remove_randomly(pct=0.6)
    
    print('\nTrial', n+1, 'of', n_trials)
    smlmc = mogptk.SM_LMC(smlmc_dataset, Q=Q)
    smlmc.init_parameters(init_method)
    try:
        smlmc.train(method=method, lr=lr, iters=iters, verbose=True, error='MSE')
    except mogptk.CholeskyException:
        smlmc_mape[n,:] = 100.0
        continue
    finally:
        print('=' * 50)

    error = mogptk.error(smlmc, per_channel=True)[0]
    smlmc_mae[n,:] = np.array([item['MAE'] for item in error])
    smlmc_rmse[n,:] = np.array([item['RMSE'] for item in error])
    smlmc_mape[n,:] = np.array([item['MAPE'] for item in error])
    smlmc_models[n] = smlmc
Trial 1 of 5

Starting optimization using Adam
‣ Model: SM-LMC
‣ Channels: 4
‣ Parameters: 22
‣ Training points: 380
‣ Initial loss: 231.985
‣ Initial error: 30883.3

Start Adam:
    0/500   0:00:00  loss=     231.985  error=     30883.3
    5/500   0:00:00  loss=     207.422  error=     26259.6
   10/500   0:00:00  loss=     192.692  error=     20356.1
   15/500   0:00:01  loss=     171.635  error=     15421.5
   20/500   0:00:01  loss=     153.482  error=     12248.2
   25/500   0:00:01  loss=     132.298  error=     11852.8
   30/500   0:00:02  loss=     113.645  error=     13247.4
   35/500   0:00:02  loss=     100.585  error=     13006.6
   40/500   0:00:02  loss=     86.9816  error=     12782.9
   45/500   0:00:03  loss=     76.1466  error=       12452
   50/500   0:00:03  loss=     66.4479  error=     13304.5
   55/500   0:00:03  loss=     58.3585  error=     13215.1
   60/500   0:00:04  loss=     52.0272  error=       13460
   65/500   0:00:04  loss=     46.7951  error=     13878.2
   70/500   0:00:04  loss=     42.2277  error=     13979.9
   75/500   0:00:05  loss=     38.3483  error=     14374.6
   80/500   0:00:05  loss=     34.9746  error=       14727
   85/500   0:00:06  loss=     31.9241  error=     15069.5
   90/500   0:00:06  loss=     29.1396  error=     15509.3
   95/500   0:00:06  loss=     26.6007  error=     15915.1
  100/500   0:00:07  loss=     24.2933  error=     16332.1
  105/500   0:00:07  loss=     22.1998  error=       16729
  110/500   0:00:08  loss=     20.3186  error=     17088.1
  115/500   0:00:08  loss=     18.6545  error=     17432.5
  120/500   0:00:09  loss=     17.1991  error=     17772.8
  125/500   0:00:09  loss=     15.9263  error=     18095.3
  130/500   0:00:09  loss=     14.7893  error=     18408.4
  135/500   0:00:10  loss=     13.7217  error=     18703.5
  140/500   0:00:10  loss=     12.6405  error=     18968.9
  145/500   0:00:10  loss=     11.4411  error=     19214.4
  150/500   0:00:11  loss=      9.9868  error=     19443.4
  155/500   0:00:11  loss=         8.1  error=     19655.6
  160/500   0:00:11  loss=     5.51175  error=     19803.6
  165/500   0:00:12  loss=      1.9654  error=     19813.2
  170/500   0:00:12  loss=    -2.36627  error=     19685.1
  175/500   0:00:12  loss=    -6.77767  error=     19756.7
  180/500   0:00:13  loss=    -10.7017  error=     19773.2
  185/500   0:00:13  loss=    -13.8503  error=     19828.3
  190/500   0:00:14  loss=     -16.204  error=     19985.1
  195/500   0:00:14  loss=    -17.9095  error=     20207.3
  200/500   0:00:14  loss=    -19.1855  error=     20440.2
  205/500   0:00:15  loss=    -20.2031  error=     20700.1
  210/500   0:00:15  loss=    -21.0621  error=     20979.3
  215/500   0:00:15  loss=    -21.8069  error=     21260.5
  220/500   0:00:16  loss=    -22.4506  error=     21514.3
  225/500   0:00:16  loss=    -22.9952  error=     21728.2
  230/500   0:00:17  loss=    -23.4453  error=     21910.5
  235/500   0:00:17  loss=    -23.8124  error=     22059.9
  240/500   0:00:17  loss=     -24.112  error=     22168.8
  245/500   0:00:18  loss=    -24.3606  error=     22240.8
  250/500   0:00:18  loss=    -24.5725  error=     22286.6
  255/500   0:00:18  loss=    -24.7593  error=     22311.3
  260/500   0:00:19  loss=    -24.9291  error=     22317.7
  265/500   0:00:19  loss=    -25.0874  error=     22309.4
  270/500   0:00:19  loss=    -25.2379  error=     22291.6
  275/500   0:00:20  loss=     -25.383  error=     22268.2
  280/500   0:00:20  loss=    -25.5244  error=     22240.9
  285/500   0:00:20  loss=     -25.664  error=     22211.3
  290/500   0:00:21  loss=    -25.8036  error=     22181.2
  295/500   0:00:21  loss=    -25.9467  error=     22151.6
  300/500   0:00:22  loss=    -26.1007  error=     22121.2
  305/500   0:00:22  loss=    -26.2985  error=     22081.3
  310/500   0:00:22  loss=    -26.8673  error=     21949.8
  315/500   0:00:23  loss=    -27.7061  error=     21668.7
  320/500   0:00:23  loss=    -28.7115  error=     21820.4
  325/500   0:00:23  loss=    -30.6874  error=     21796.8
  330/500   0:00:24  loss=    -34.3587  error=     21812.9
  335/500   0:00:24  loss=    -38.3639  error=     22027.7
  340/500   0:00:24  loss=    -42.0415  error=     22522.8
  345/500   0:00:25  loss=    -45.0274  error=     22940.9
  350/500   0:00:25  loss=    -47.1926  error=     23129.7
  355/500   0:00:26  loss=    -48.6912  error=     23294.9
  360/500   0:00:26  loss=    -49.7321  error=     23423.8
  365/500   0:00:26  loss=    -50.4937  error=     23429.8
  370/500   0:00:27  loss=    -51.1098  error=       23344
  375/500   0:00:27  loss=    -51.6622  error=     23280.3
  380/500   0:00:27  loss=    -52.1912  error=     23215.6
  385/500   0:00:28  loss=    -52.7093  error=     23114.8
  390/500   0:00:28  loss=    -53.2127  error=     23019.9
  395/500   0:00:29  loss=    -53.6956  error=     22941.8
  400/500   0:00:29  loss=    -54.1533  error=     22858.8
  405/500   0:00:29  loss=    -54.5836  error=       22769
  410/500   0:00:30  loss=    -54.9857  error=     22688.1
  415/500   0:00:30  loss=    -55.3594  error=     22615.3
  420/500   0:00:30  loss=    -55.7048  error=     22544.4
  425/500   0:00:31  loss=    -56.0235  error=     22480.9
  430/500   0:00:31  loss=    -56.3188  error=     22425.8
  435/500   0:00:32  loss=    -56.5986  error=     22371.4
  440/500   0:00:32  loss=    -56.8813  error=     22312.6
  445/500   0:00:32  loss=    -57.2139  error=     22240.3
  450/500   0:00:33  loss=    -57.6762  error=     22137.1
  455/500   0:00:33  loss=    -58.2343  error=     22040.3
  460/500   0:00:33  loss=    -58.8041  error=     22005.8
  465/500   0:00:34  loss=    -59.2921  error=     21969.9
  470/500   0:00:34  loss=    -59.7215  error=     21956.9
  475/500   0:00:35  loss=    -60.1494  error=     21969.8
  480/500   0:00:35  loss=    -60.6924  error=     21973.2
  485/500   0:00:35  loss=    -61.8514  error=     21925.4
  490/500   0:00:36  loss=    -65.2454  error=     21549.5
  495/500   0:00:36  loss=    -70.8874  error=     21662.2
  500/500   0:00:36  loss=    -76.0655  error=     21715.7
Finished

Optimization finished in 36.858 seconds
‣ Iterations: 500
‣ Final loss: -76.0655
‣ Final error: 21715.7
==================================================

Trial 2 of 5

Starting optimization using Adam
‣ Model: SM-LMC
‣ Channels: 4
‣ Parameters: 22
‣ Training points: 382
‣ Initial loss: 209.867
‣ Initial error: 35687.6

Start Adam:
    0/500   0:00:00  loss=     209.867  error=     35687.6
    5/500   0:00:00  loss=      184.35  error=       32872
   10/500   0:00:00  loss=     165.394  error=       29388
   15/500   0:00:01  loss=     135.101  error=     25392.7
   20/500   0:00:01  loss=     117.059  error=     19301.7
   25/500   0:00:01  loss=     99.5664  error=     14782.6
   30/500   0:00:02  loss=     83.7732  error=     13805.8
   35/500   0:00:02  loss=     71.9856  error=     13612.8
   40/500   0:00:02  loss=     62.4483  error=     13664.4
   45/500   0:00:03  loss=     54.7018  error=     13700.2
   50/500   0:00:03  loss=     48.3957  error=       13478
   55/500   0:00:03  loss=     42.6044  error=     13425.3
   60/500   0:00:04  loss=     37.9309  error=       13342
   65/500   0:00:04  loss=     33.8613  error=     13492.7
   70/500   0:00:05  loss=     30.2727  error=     13698.7
   75/500   0:00:05  loss=     27.1455  error=     13759.9
   80/500   0:00:05  loss=     24.4496  error=     13839.6
   85/500   0:00:06  loss=     22.1026  error=     14037.9
   90/500   0:00:06  loss=     20.0809  error=       14186
   95/500   0:00:06  loss=     18.3576  error=     14413.5
  100/500   0:00:07  loss=     16.9061  error=     14696.4
  105/500   0:00:07  loss=     15.6961  error=     14947.7
  110/500   0:00:07  loss=     14.6892  error=     15202.1
  115/500   0:00:08  loss=     13.8459  error=     15454.7
  120/500   0:00:08  loss=     13.1245  error=       15698
  125/500   0:00:09  loss=     12.4861  error=     15930.2
  130/500   0:00:09  loss=      11.894  error=       16162
  135/500   0:00:09  loss=     11.3081  error=     16379.4
  140/500   0:00:10  loss=     10.6715  error=     16587.5
  145/500   0:00:10  loss=     9.88242  error=     16780.4
  150/500   0:00:11  loss=     8.75515  error=     16942.9
  155/500   0:00:11  loss=     7.05657  error=     17087.6
  160/500   0:00:11  loss=     4.64124  error=     17230.5
  165/500   0:00:12  loss=     1.52978  error=     17387.8
  170/500   0:00:12  loss=    -2.07251  error=     17559.8
  175/500   0:00:12  loss=    -5.78227  error=     17769.2
  180/500   0:00:13  loss=    -9.18281  error=     18038.3
  185/500   0:00:13  loss=    -12.0153  error=     18332.9
  190/500   0:00:14  loss=    -14.2198  error=     18618.2
  195/500   0:00:14  loss=    -15.8658  error=       18869
  200/500   0:00:14  loss=    -17.0714  error=     19099.8
  205/500   0:00:15  loss=    -17.9602  error=     19328.2
  210/500   0:00:15  loss=    -18.6418  error=     19560.3
  215/500   0:00:15  loss=    -19.1996  error=     19794.6
  220/500   0:00:16  loss=    -19.6874  error=       20023
  225/500   0:00:16  loss=    -20.1342  error=     20238.3
  230/500   0:00:17  loss=    -20.5533  error=       20436
  235/500   0:00:17  loss=      -20.95  error=     20613.5
  240/500   0:00:17  loss=    -21.3258  error=     20768.3
  245/500   0:00:18  loss=    -21.6804  error=     20898.7
  250/500   0:00:18  loss=    -22.0128  error=     21003.3
  255/500   0:00:18  loss=    -22.3218  error=     21081.6
  260/500   0:00:19  loss=    -22.6063  error=     21135.2
  265/500   0:00:19  loss=    -22.8661  error=     21167.7
  270/500   0:00:20  loss=    -23.1021  error=     21184.4
  275/500   0:00:20  loss=     -23.316  error=     21190.5
  280/500   0:00:20  loss=    -23.5103  error=     21190.9
  285/500   0:00:21  loss=    -23.6879  error=     21189.8
  290/500   0:00:21  loss=    -23.8516  error=     21189.9
  295/500   0:00:21  loss=    -24.0036  error=     21193.6
  300/500   0:00:22  loss=    -24.1461  error=     21201.9
  305/500   0:00:22  loss=    -24.2805  error=     21215.4
  310/500   0:00:23  loss=     -24.408  error=     21234.3
  315/500   0:00:23  loss=    -24.5297  error=     21258.1
  320/500   0:00:23  loss=     -24.646  error=     21286.4
  325/500   0:00:24  loss=    -24.7575  error=     21318.4
  330/500   0:00:24  loss=    -24.8646  error=     21353.3
  335/500   0:00:24  loss=    -24.9676  error=     21390.5
  340/500   0:00:25  loss=    -25.0669  error=     21429.1
  345/500   0:00:25  loss=    -25.1625  error=     21468.5
  350/500   0:00:25  loss=    -25.2548  error=     21508.2
  355/500   0:00:26  loss=     -25.344  error=     21547.8
  360/500   0:00:26  loss=    -25.4301  error=     21586.9
  365/500   0:00:26  loss=    -25.5133  error=     21625.3
  370/500   0:00:27  loss=    -25.5938  error=     21662.8
  375/500   0:00:27  loss=    -25.6717  error=     21699.3
  380/500   0:00:28  loss=     -25.747  error=     21734.7
  385/500   0:00:28  loss=    -25.8198  error=       21769
  390/500   0:00:28  loss=    -25.8903  error=     21802.2
  395/500   0:00:29  loss=    -25.9586  error=     21834.1
  400/500   0:00:29  loss=    -26.0246  error=     21864.8
  405/500   0:00:29  loss=    -26.0885  error=     21894.2
  410/500   0:00:30  loss=    -26.1504  error=     21922.4
  415/500   0:00:30  loss=    -26.2102  error=     21949.3
  420/500   0:00:30  loss=    -26.2682  error=       21975
  425/500   0:00:31  loss=    -26.3244  error=     21999.5
  430/500   0:00:31  loss=    -26.3788  error=     22022.8
  435/500   0:00:32  loss=    -26.4315  error=     22044.9
  440/500   0:00:32  loss=    -26.4826  error=     22065.9
  445/500   0:00:32  loss=    -26.5321  error=     22085.9
  450/500   0:00:33  loss=    -26.5801  error=     22104.7
  455/500   0:00:33  loss=    -26.6268  error=     22122.6
  460/500   0:00:33  loss=    -26.6721  error=     22139.6
  465/500   0:00:34  loss=    -26.7161  error=     22155.7
  470/500   0:00:34  loss=    -26.7589  error=       22171
  475/500   0:00:35  loss=    -26.8005  error=     22185.5
  480/500   0:00:35  loss=     -26.841  error=     22199.3
  485/500   0:00:35  loss=    -26.8806  error=     22212.4
  490/500   0:00:36  loss=    -26.9191  error=     22224.9
  495/500   0:00:36  loss=    -26.9567  error=     22236.8
  500/500   0:00:36  loss=    -26.9934  error=     22248.3
Finished

Optimization finished in 36.866 seconds
‣ Iterations: 500
‣ Final loss: -26.9934
‣ Final error: 22248.3
==================================================

Trial 3 of 5

Starting optimization using Adam
‣ Model: SM-LMC
‣ Channels: 4
‣ Parameters: 22
‣ Training points: 377
‣ Initial loss: 213.502
‣ Initial error: 25236.4

Start Adam:
    0/500   0:00:00  loss=     213.502  error=     25236.4
    5/500   0:00:00  loss=     188.314  error=     23300.1
   10/500   0:00:00  loss=     176.189  error=     17250.6
   15/500   0:00:01  loss=     154.179  error=     15982.1
   20/500   0:00:01  loss=     140.504  error=     14096.9
   25/500   0:00:02  loss=     127.579  error=     13372.1
   30/500   0:00:03  loss=     115.535  error=     12911.8
   35/500   0:00:03  loss=     105.501  error=     12410.4
   40/500   0:00:04  loss=     95.3847  error=     12350.1
   45/500   0:00:04  loss=     86.4981  error=       12101
   50/500   0:00:05  loss=     78.1167  error=     13253.6
   55/500   0:00:05  loss=     68.8057  error=     14597.1
   60/500   0:00:06  loss=      59.835  error=     13221.6
   65/500   0:00:06  loss=     52.5518  error=     13000.7
   70/500   0:00:07  loss=     46.5633  error=     13151.3
   75/500   0:00:08  loss=      41.172  error=     13186.1
   80/500   0:00:08  loss=     36.5343  error=     13326.2
   85/500   0:00:09  loss=      32.424  error=     13334.3
   90/500   0:00:09  loss=     28.7358  error=       13238
   95/500   0:00:09  loss=     25.3539  error=     13159.7
  100/500   0:00:10  loss=     22.3559  error=     13186.2
  105/500   0:00:11  loss=     19.7167  error=     13343.2
  110/500   0:00:11  loss=     17.4505  error=       13547
  115/500   0:00:12  loss=     15.4862  error=     13696.7
  120/500   0:00:12  loss=     13.7624  error=     13853.3
  125/500   0:00:12  loss=     12.2319  error=       14061
  130/500   0:00:13  loss=     10.8658  error=     14299.6
  135/500   0:00:13  loss=     9.64198  error=     14564.9
  140/500   0:00:14  loss=     8.54071  error=     14863.5
  145/500   0:00:14  loss=     7.54095  error=     15189.2
  150/500   0:00:15  loss=     6.62226  error=     15534.9
  155/500   0:00:15  loss=     5.76736  error=     15901.7
  160/500   0:00:15  loss=     4.96327  error=     16286.5
  165/500   0:00:16  loss=     4.20313  error=     16674.4
  170/500   0:00:16  loss=     3.48722  error=       17050
  175/500   0:00:17  loss=     2.81954  error=     17392.6
  180/500   0:00:17  loss=     2.20116  error=     17684.9
  185/500   0:00:18  loss=     1.62938  error=     17925.8
  190/500   0:00:18  loss=     1.09164  error=     18119.4
  195/500   0:00:19  loss=    0.561379  error=     18259.1
  200/500   0:00:19  loss=  -0.0148752  error=     18329.5
  205/500   0:00:19  loss=   -0.748273  error=     18314.7
  210/500   0:00:20  loss=    -1.86039  error=     18192.7
  215/500   0:00:20  loss=    -3.64044  error=     17975.6
  220/500   0:00:21  loss=    -6.16145  error=     17800.5
  225/500   0:00:22  loss=     -9.1421  error=     17799.8
  230/500   0:00:23  loss=     -12.107  error=     17959.5
  235/500   0:00:23  loss=    -14.6812  error=     18164.6
  240/500   0:00:24  loss=    -16.7537  error=     18358.6
  245/500   0:00:24  loss=    -18.3915  error=     18581.8
  250/500   0:00:24  loss=    -19.7053  error=     18839.9
  255/500   0:00:25  loss=     -20.762  error=     19079.3
  260/500   0:00:25  loss=    -21.6015  error=     19266.6
  265/500   0:00:26  loss=    -22.2664  error=     19397.5
  270/500   0:00:26  loss=    -22.8035  error=       19478
  275/500   0:00:26  loss=    -23.2597  error=     19514.9
  280/500   0:00:27  loss=    -23.6908  error=     19505.5
  285/500   0:00:27  loss=     -24.229  error=     19435.6
  290/500   0:00:28  loss=    -25.4739  error=     19237.7
  295/500   0:00:28  loss=    -27.2885  error=     18926.3
  300/500   0:00:28  loss=    -29.7369  error=     18762.6
  305/500   0:00:29  loss=    -33.4294  error=     18341.3
  310/500   0:00:29  loss=     -37.085  error=       18123
  315/500   0:00:29  loss=    -39.9652  error=     17996.5
  320/500   0:00:30  loss=    -42.0127  error=     17827.4
  325/500   0:00:30  loss=    -43.4627  error=     17782.1
  330/500   0:00:30  loss=    -44.4904  error=     17661.4
  335/500   0:00:31  loss=    -45.2215  error=     17516.7
  340/500   0:00:31  loss=    -45.8205  error=     17408.2
  345/500   0:00:32  loss=    -46.3638  error=     17203.2
  350/500   0:00:32  loss=    -46.9049  error=     17000.4
  355/500   0:00:32  loss=    -47.4582  error=     16776.5
  360/500   0:00:33  loss=    -48.0322  error=     16567.4
  365/500   0:00:33  loss=    -48.6281  error=     16351.7
  370/500   0:00:33  loss=    -49.2449  error=       16138
  375/500   0:00:34  loss=     -49.878  error=     15949.6
  380/500   0:00:34  loss=    -50.5181  error=     15782.4
  385/500   0:00:34  loss=    -51.1537  error=     15633.7
  390/500   0:00:35  loss=    -51.7732  error=     15502.6
  395/500   0:00:35  loss=    -52.3664  error=     15389.1
  400/500   0:00:35  loss=    -52.9252  error=     15292.5
  405/500   0:00:36  loss=    -53.4443  error=     15210.5
  410/500   0:00:36  loss=    -53.9205  error=     15140.4
  415/500   0:00:37  loss=     -54.353  error=     15080.1
  420/500   0:00:37  loss=    -54.7421  error=     15027.7
  425/500   0:00:37  loss=    -55.0901  error=     14981.3
  430/500   0:00:38  loss=    -55.4002  error=     14938.8
  435/500   0:00:38  loss=    -55.6763  error=     14899.2
  440/500   0:00:38  loss=    -55.9231  error=     14861.8
  445/500   0:00:39  loss=    -56.1449  error=     14825.9
  450/500   0:00:39  loss=    -56.3459  error=     14791.3
  455/500   0:00:39  loss=    -56.5295  error=     14757.8
  460/500   0:00:40  loss=    -56.6989  error=     14725.2
  465/500   0:00:40  loss=    -56.8565  error=     14693.8
  470/500   0:00:40  loss=    -57.0043  error=     14663.3
  475/500   0:00:41  loss=    -57.1437  error=     14633.9
  480/500   0:00:41  loss=    -57.2762  error=     14605.6
  485/500   0:00:41  loss=    -57.4025  error=     14578.2
  490/500   0:00:42  loss=    -57.5235  error=     14551.8
  495/500   0:00:42  loss=    -57.6398  error=     14526.3
  500/500   0:00:43  loss=    -57.7519  error=     14501.8
Finished

Optimization finished in 43.222 seconds
‣ Iterations: 500
‣ Final loss: -57.7519
‣ Final error: 14501.8
==================================================

Trial 4 of 5

Starting optimization using Adam
‣ Model: SM-LMC
‣ Channels: 4
‣ Parameters: 22
‣ Training points: 385
‣ Initial loss: 228.005
‣ Initial error: 36893.6

Start Adam:
    0/500   0:00:00  loss=     228.005  error=     36893.6
    5/500   0:00:00  loss=     198.195  error=     21519.8
   10/500   0:00:00  loss=     177.419  error=     14116.9
   15/500   0:00:01  loss=     151.815  error=     14888.4
   20/500   0:00:01  loss=     128.466  error=     13757.8
   25/500   0:00:01  loss=     108.461  error=       15208
   30/500   0:00:02  loss=     94.8166  error=     15218.8
   35/500   0:00:02  loss=     80.9806  error=     14683.4
   40/500   0:00:02  loss=     67.8589  error=     14030.7
   45/500   0:00:03  loss=     56.2273  error=     13923.9
   50/500   0:00:03  loss=     46.6471  error=     13549.3
   55/500   0:00:04  loss=      38.983  error=     13531.1
   60/500   0:00:04  loss=     32.8355  error=       13767
   65/500   0:00:04  loss=     28.0891  error=     13748.8
   70/500   0:00:05  loss=     24.4794  error=     13945.1
   75/500   0:00:05  loss=     21.6842  error=     13959.2
   80/500   0:00:05  loss=     19.5094  error=     14079.5
   85/500   0:00:06  loss=     17.7923  error=     14150.9
   90/500   0:00:06  loss=      16.394  error=       14306
   95/500   0:00:06  loss=     15.2322  error=     14428.6
  100/500   0:00:07  loss=     14.2483  error=     14602.2
  105/500   0:00:07  loss=     13.3831  error=     14732.5
  110/500   0:00:07  loss=     12.5773  error=     14890.1
  115/500   0:00:08  loss=     11.7703  error=     15014.5
  120/500   0:00:08  loss=     10.8872  error=     15150.3
  125/500   0:00:09  loss=     9.84518  error=     15253.7
  130/500   0:00:09  loss=     8.57297  error=     15349.3
  135/500   0:00:09  loss=     7.02585  error=     15418.1
  140/500   0:00:10  loss=     5.19973  error=     15480.2
  145/500   0:00:10  loss=     3.15594  error=     15552.2
  150/500   0:00:10  loss=     1.04436  error=     15660.7
  155/500   0:00:11  loss=   -0.930201  error=     15821.2
  160/500   0:00:11  loss=    -2.60926  error=     16025.6
  165/500   0:00:11  loss=    -3.95194  error=     16264.9
  170/500   0:00:12  loss=    -5.01579  error=     16523.1
  175/500   0:00:12  loss=    -5.89447  error=     16793.1
  180/500   0:00:12  loss=    -6.67165  error=       17067
  185/500   0:00:13  loss=    -7.40606  error=     17340.7
  190/500   0:00:13  loss=     -8.1341  error=     17608.5
  195/500   0:00:14  loss=    -8.87649  error=     17865.8
  200/500   0:00:14  loss=    -9.64367  error=     18108.6
  205/500   0:00:14  loss=    -10.4394  error=       18334
  210/500   0:00:15  loss=    -11.2633  error=     18540.3
  215/500   0:00:15  loss=    -12.1129  error=     18726.4
  220/500   0:00:15  loss=    -12.9852  error=     18891.1
  225/500   0:00:16  loss=    -13.8778  error=     19032.7
  230/500   0:00:16  loss=    -14.7889  error=     19150.1
  235/500   0:00:16  loss=    -15.7138  error=     19246.3
  240/500   0:00:17  loss=    -16.6385  error=     19332.5
  245/500   0:00:17  loss=    -17.5378  error=     19419.4
  250/500   0:00:18  loss=    -18.3862  error=     19506.9
  255/500   0:00:18  loss=    -19.1714  error=     19591.3
  260/500   0:00:18  loss=    -19.8967  error=       19672
  265/500   0:00:19  loss=    -20.5777  error=     19749.4
  270/500   0:00:19  loss=    -21.2377  error=     19825.3
  275/500   0:00:19  loss=    -21.8926  error=     19902.4
  280/500   0:00:20  loss=    -22.5287  error=     19982.7
  285/500   0:00:20  loss=    -23.1221  error=     20066.7
  290/500   0:00:20  loss=    -23.6715  error=     20150.7
  295/500   0:00:21  loss=    -24.1936  error=     20226.3
  300/500   0:00:21  loss=    -24.7263  error=       20288
  305/500   0:00:21  loss=    -25.4832  error=     20335.2
  310/500   0:00:22  loss=     -27.519  error=     20345.9
  315/500   0:00:22  loss=    -30.6456  error=       20222
  320/500   0:00:23  loss=    -35.1424  error=     20120.5
  325/500   0:00:23  loss=    -39.8357  error=     20013.7
  330/500   0:00:23  loss=    -44.0443  error=     19977.6
  335/500   0:00:24  loss=    -47.4904  error=     19984.7
  340/500   0:00:24  loss=    -50.0902  error=     20001.7
  345/500   0:00:24  loss=    -51.9791  error=     20069.8
  350/500   0:00:25  loss=    -53.3177  error=     20165.1
  355/500   0:00:25  loss=    -54.3079  error=     20297.2
  360/500   0:00:25  loss=    -55.1102  error=     20426.2
  365/500   0:00:26  loss=    -55.8233  error=     20533.4
  370/500   0:00:26  loss=    -56.4816  error=     20629.4
  375/500   0:00:26  loss=    -57.0834  error=     20716.7
  380/500   0:00:27  loss=    -57.6164  error=     20788.7
  385/500   0:00:27  loss=    -58.0746  error=     20843.9
  390/500   0:00:28  loss=    -58.4614  error=       20885
  395/500   0:00:28  loss=    -58.7858  error=     20913.1
  400/500   0:00:28  loss=    -59.0589  error=       20929
  405/500   0:00:29  loss=    -59.2909  error=     20935.8
  410/500   0:00:29  loss=    -59.4903  error=     20936.5
  415/500   0:00:29  loss=    -59.6639  error=     20930.9
  420/500   0:00:30  loss=     -59.817  error=     20919.4
  425/500   0:00:30  loss=    -59.9538  error=     20904.1
  430/500   0:00:30  loss=    -60.0777  error=     20887.2
  435/500   0:00:31  loss=    -60.1914  error=     20869.1
  440/500   0:00:31  loss=     -60.297  error=     20849.5
  445/500   0:00:31  loss=     -60.396  error=     20828.7
  450/500   0:00:32  loss=    -60.4895  error=     20807.5
  455/500   0:00:32  loss=    -60.5785  error=     20786.4
  460/500   0:00:33  loss=    -60.6634  error=     20764.9
  465/500   0:00:33  loss=    -60.7449  error=     20742.9
  470/500   0:00:33  loss=    -60.8233  error=     20720.5
  475/500   0:00:34  loss=    -60.8989  error=     20697.9
  480/500   0:00:34  loss=     -60.972  error=       20675
  485/500   0:00:34  loss=    -61.0428  error=     20651.9
  490/500   0:00:35  loss=    -61.1115  error=     20628.5
  495/500   0:00:35  loss=    -61.1783  error=     20604.9
  500/500   0:00:35  loss=    -61.2432  error=     20581.4
Finished

Optimization finished in 35.944 seconds
‣ Iterations: 500
‣ Final loss: -61.2432
‣ Final error: 20581.4
==================================================

Trial 5 of 5

Starting optimization using Adam
‣ Model: SM-LMC
‣ Channels: 4
‣ Parameters: 22
‣ Training points: 383
‣ Initial loss: 215.95
‣ Initial error: 35865.2

Start Adam:
    0/500   0:00:00  loss=      215.95  error=     35865.2
    5/500   0:00:00  loss=     191.617  error=     25759.6
   10/500   0:00:00  loss=     170.826  error=     21906.2
   15/500   0:00:01  loss=     140.981  error=     14889.8
   20/500   0:00:01  loss=     128.202  error=     15301.2
   25/500   0:00:01  loss=     114.187  error=     16297.5
   30/500   0:00:02  loss=     102.384  error=     16275.7
   35/500   0:00:02  loss=     90.5687  error=     16319.4
   40/500   0:00:02  loss=     78.5556  error=       18196
   45/500   0:00:03  loss=     68.0478  error=       18737
   50/500   0:00:03  loss=     57.0045  error=     19057.1
   55/500   0:00:03  loss=     47.1645  error=       19355
   60/500   0:00:04  loss=     38.3969  error=     18865.8
   65/500   0:00:04  loss=     30.7794  error=       18823
   70/500   0:00:05  loss=     24.6679  error=     18856.5
   75/500   0:00:05  loss=     19.6544  error=       18880
   80/500   0:00:05  loss=     15.6079  error=     19236.9
   85/500   0:00:06  loss=     12.2842  error=     19630.9
   90/500   0:00:06  loss=     9.43567  error=     20064.6
   95/500   0:00:06  loss=     6.94086  error=       20546
  100/500   0:00:07  loss=     4.63802  error=     21078.6
  105/500   0:00:07  loss=     2.32316  error=       21544
  110/500   0:00:07  loss=   -0.234247  error=     21942.7
  115/500   0:00:08  loss=    -3.21631  error=     22338.6
  120/500   0:00:08  loss=    -6.72422  error=     22688.3
  125/500   0:00:08  loss=    -10.7596  error=     22963.8
  130/500   0:00:09  loss=    -15.1791  error=     23192.8
  135/500   0:00:09  loss=    -19.6795  error=     23400.2
  140/500   0:00:10  loss=    -23.9138  error=     23628.5
  145/500   0:00:10  loss=    -27.6493  error=     23913.4
  150/500   0:00:10  loss=    -30.8142  error=     24242.3
  155/500   0:00:11  loss=    -33.4389  error=     24589.9
  160/500   0:00:11  loss=    -35.5902  error=       24927
  165/500   0:00:11  loss=    -37.3431  error=     25234.3
  170/500   0:00:12  loss=     -38.774  error=       25498
  175/500   0:00:12  loss=    -39.9561  error=     25708.5
  180/500   0:00:12  loss=    -40.9545  error=     25857.9
  185/500   0:00:13  loss=    -41.8212  error=     25941.4
  190/500   0:00:13  loss=    -42.5931  error=     25960.8
  195/500   0:00:14  loss=    -43.2929  error=     25924.3
  200/500   0:00:14  loss=    -43.9323  error=     25844.7
  205/500   0:00:14  loss=    -44.5169  error=     25737.6
  210/500   0:00:15  loss=    -45.0496  error=     25617.9
  215/500   0:00:15  loss=    -45.5332  error=     25497.8
  220/500   0:00:16  loss=    -45.9705  error=     25384.6
  225/500   0:00:16  loss=    -46.3643  error=     25281.3
  230/500   0:00:16  loss=    -46.7174  error=       25187
  235/500   0:00:17  loss=    -47.0331  error=     25099.3
  240/500   0:00:17  loss=    -47.3149  error=     25016.2
  245/500   0:00:17  loss=    -47.5665  error=     24937.3
  250/500   0:00:18  loss=    -47.7914  error=     24863.3
  255/500   0:00:18  loss=    -47.9928  error=     24795.9
  260/500   0:00:19  loss=    -48.1738  error=       24736
  265/500   0:00:19  loss=    -48.3369  error=     24683.4
  270/500   0:00:19  loss=    -48.4847  error=     24637.3
  275/500   0:00:20  loss=    -48.6192  error=     24596.4
  280/500   0:00:20  loss=    -48.7424  error=     24559.6
  285/500   0:00:20  loss=    -48.8558  error=     24526.5
  290/500   0:00:21  loss=    -48.9609  error=       24497
  295/500   0:00:21  loss=     -49.059  error=     24470.9
  300/500   0:00:21  loss=    -49.1511  error=     24448.1
  305/500   0:00:22  loss=    -49.2383  error=     24428.3
  310/500   0:00:22  loss=    -49.3213  error=     24411.1
  315/500   0:00:23  loss=    -49.4009  error=     24396.2
  320/500   0:00:23  loss=    -49.4778  error=     24383.3
  325/500   0:00:23  loss=    -49.5526  error=     24372.4
  330/500   0:00:24  loss=    -49.6259  error=     24363.4
  335/500   0:00:24  loss=    -49.6981  error=     24356.1
  340/500   0:00:24  loss=    -49.7699  error=     24350.6
  345/500   0:00:25  loss=    -49.8417  error=     24346.5
  350/500   0:00:25  loss=     -49.914  error=       24344
  355/500   0:00:25  loss=    -49.9874  error=     24342.9
  360/500   0:00:26  loss=    -50.0627  error=     24343.2
  365/500   0:00:26  loss=    -50.1406  error=       24345
  370/500   0:00:26  loss=    -50.2222  error=     24348.3
  375/500   0:00:27  loss=    -50.3089  error=     24353.2
  380/500   0:00:27  loss=    -50.4026  error=       24360
  385/500   0:00:28  loss=    -50.5062  error=     24368.9
  390/500   0:00:28  loss=    -50.6235  error=     24380.4
  395/500   0:00:28  loss=    -50.7592  error=     24394.7
  400/500   0:00:29  loss=    -50.9167  error=     24412.1
  405/500   0:00:29  loss=     -51.094  error=     24432.3
  410/500   0:00:29  loss=    -51.2845  error=     24455.1
  415/500   0:00:30  loss=    -51.4848  error=     24479.3
  420/500   0:00:30  loss=    -51.6937  error=     24500.9
  425/500   0:00:31  loss=    -51.9088  error=     24514.3
  430/500   0:00:31  loss=    -52.1316  error=     24517.4
  435/500   0:00:31  loss=    -52.3666  error=     24511.9
  440/500   0:00:32  loss=    -52.6207  error=     24499.3
  445/500   0:00:32  loss=    -52.9028  error=     24478.4
  450/500   0:00:32  loss=    -53.2215  error=     24446.7
  455/500   0:00:33  loss=    -53.5774  error=     24404.8
  460/500   0:00:33  loss=    -53.9598  error=     24359.7
  465/500   0:00:33  loss=    -54.3641  error=     24316.6
  470/500   0:00:34  loss=    -54.7892  error=     24267.6
  475/500   0:00:34  loss=    -55.2318  error=     24198.8
  480/500   0:00:34  loss=    -55.6977  error=     24104.7
  485/500   0:00:35  loss=    -56.1981  error=     23987.3
  490/500   0:00:35  loss=    -56.7438  error=       23849
  495/500   0:00:36  loss=    -57.3356  error=     23690.2
  500/500   0:00:36  loss=    -57.9558  error=     23512.5
Finished

Optimization finished in 36.412 seconds
‣ Iterations: 500
‣ Final loss: -57.9558
‣ Final error: 23512.5
==================================================
In [12]:
pd.DataFrame(np.c_[smlmc_mae.mean(1), smlmc_rmse.mean(1), smlmc_mape.mean(1),
             smlmc_mae.std(1), smlmc_rmse.std(1), smlmc_mape.std(1)],
             columns=['MAE', 'RMSE', 'MAPE', 'MAE std', 'RMSE std', 'MAPE std'])
Out[12]:
MAE RMSE MAPE MAE std RMSE std MAPE std
0 63.644562 76.823941 2.612349 98.284175 118.078722 1.411013
1 59.997056 80.053514 3.048242 87.743939 117.413253 1.686338
2 52.329180 66.387327 2.527252 74.883674 94.295961 1.081837
3 66.393357 77.838839 2.956780 96.820934 112.255519 1.396803
4 67.441888 82.688507 3.438434 100.536282 120.393691 2.147719
In [13]:
best_smlmc = smlmc_models[np.argmin(smlmc_mape.mean(1))]
best_smlmc.plot_prediction();

Independent spectral mixture kernels¶

In [14]:
sm_models = [None] * n_trials
sm_mae = np.zeros((n_trials,4))
sm_rmse = np.zeros((n_trials,4))
sm_mape = np.zeros((n_trials,4))

for n in range(n_trials):
    sm_dataset = dataset.copy()
    sm_dataset['Gold'].remove_randomly(pct=0.6)
    sm_dataset['Oil'].remove_randomly(pct=0.3)
    sm_dataset['NASDAQ'].remove_randomly(pct=0.6)
    sm_dataset['USD'].remove_randomly(pct=0.6)
    
    print('\nTrial', n+1, 'of', n_trials)
    sm = mogptk.SM(sm_dataset, Q=Q)
    sm.init_parameters(init_method)
    try:
        sm.train(method=method, lr=lr, iters=iters, verbose=True, error='MSE')
    except mogptk.CholeskyException:
        sm_mape[n,:] = 100.0
        continue
    finally:
        print('=' * 50)
        
    error = mogptk.error(sm, per_channel=True)[0]
    sm_mae[n,:] = np.array([item['MAE'] for item in error])
    sm_rmse[n,:] = np.array([item['RMSE'] for item in error])
    sm_mape[n,:] = np.array([item['MAPE'] for item in error])
    sm_models[n] = sm
Trial 1 of 5

Starting optimization using Adam
‣ Model: SM
‣ Channels: 4
‣ Parameters: 40
‣ Training points: 381
‣ Initial loss: 129.982
‣ Initial error: 12099.4

Start Adam:
    0/500   0:00:00  loss=     129.982  error=     12099.4
    5/500   0:00:00  loss=     111.776  error=     13294.2
   10/500   0:00:00  loss=     97.5277  error=     15494.4
   15/500   0:00:00  loss=     83.8153  error=       15829
   20/500   0:00:00  loss=     70.5513  error=       16110
   25/500   0:00:00  loss=     57.8898  error=     16299.8
   30/500   0:00:01  loss=     45.8837  error=     16507.2
   35/500   0:00:01  loss=     34.5061  error=     16649.7
   40/500   0:00:01  loss=     23.8687  error=     16696.2
   45/500   0:00:01  loss=     13.9986  error=     16640.5
   50/500   0:00:01  loss=     4.85495  error=     16401.6
   55/500   0:00:01  loss=    -4.21225  error=     14674.7
   60/500   0:00:02  loss=    -13.1147  error=     12645.5
   65/500   0:00:02  loss=    -20.7552  error=       12785
   70/500   0:00:02  loss=    -27.2975  error=     12485.9
   75/500   0:00:02  loss=    -33.1166  error=     12553.6
   80/500   0:00:02  loss=    -38.2423  error=     12370.2
   85/500   0:00:02  loss=    -42.7618  error=     12400.6
   90/500   0:00:03  loss=    -46.6479  error=       12293
   95/500   0:00:03  loss=    -49.9714  error=     12273.8
  100/500   0:00:03  loss=    -52.8401  error=     12222.8
  105/500   0:00:03  loss=    -55.2752  error=     12168.8
  110/500   0:00:03  loss=    -57.3274  error=     12150.8
  115/500   0:00:03  loss=      -59.03  error=     12091.3
  120/500   0:00:04  loss=    -60.4432  error=     12076.8
  125/500   0:00:04  loss=    -61.8997  error=     12041.7
  130/500   0:00:04  loss=    -62.8034  error=     12012.2
  135/500   0:00:04  loss=    -64.8527  error=       11999
  140/500   0:00:04  loss=    -66.3958  error=     11974.2
  145/500   0:00:04  loss=    -68.5996  error=     11959.9
  150/500   0:00:05  loss=    -70.7324  error=     11949.8
  155/500   0:00:05  loss=    -72.6433  error=     11935.8
  160/500   0:00:05  loss=    -74.4698  error=     11928.8
  165/500   0:00:05  loss=     -77.192  error=     11922.4
  170/500   0:00:05  loss=     -81.124  error=       11915
  175/500   0:00:05  loss=    -84.9452  error=     11911.2
  180/500   0:00:06  loss=    -88.3807  error=     11906.9
  185/500   0:00:06  loss=    -91.1819  error=     11902.6
  190/500   0:00:06  loss=    -93.6131  error=     11899.6
  195/500   0:00:06  loss=    -95.8055  error=     11895.8
  200/500   0:00:06  loss=    -97.7675  error=     11892.5
  205/500   0:00:06  loss=    -99.4454  error=     11889.5
  210/500   0:00:07  loss=    -100.837  error=     11886.2
  215/500   0:00:07  loss=    -102.001  error=     11883.5
  220/500   0:00:07  loss=    -103.028  error=     11880.8
  225/500   0:00:07  loss=    -104.006  error=     11878.4
  230/500   0:00:07  loss=    -104.982  error=     11876.5
  235/500   0:00:07  loss=    -105.951  error=     11874.9
  240/500   0:00:08  loss=    -106.878  error=     11873.9
  245/500   0:00:08  loss=    -107.729  error=     11873.2
  250/500   0:00:08  loss=    -108.485  error=       11873
  255/500   0:00:08  loss=    -109.143  error=     11873.2
  260/500   0:00:08  loss=    -109.718  error=     11873.7
  265/500   0:00:08  loss=    -110.231  error=     11874.7
  270/500   0:00:09  loss=    -110.699  error=     11875.9
  275/500   0:00:09  loss=    -111.128  error=     11877.4
  280/500   0:00:09  loss=    -111.516  error=     11879.3
  285/500   0:00:09  loss=    -111.854  error=     11881.3
  290/500   0:00:09  loss=    -112.142  error=     11883.6
  295/500   0:00:09  loss=    -112.386  error=     11886.1
  300/500   0:00:10  loss=    -112.593  error=     11888.8
  305/500   0:00:10  loss=    -112.774  error=     11891.6
  310/500   0:00:10  loss=    -112.936  error=     11894.6
  315/500   0:00:10  loss=    -113.082  error=     11897.6
  320/500   0:00:10  loss=    -113.217  error=     11900.8
  325/500   0:00:10  loss=    -113.342  error=       11904
  330/500   0:00:11  loss=    -113.461  error=     11907.3
  335/500   0:00:11  loss=    -113.574  error=     11910.7
  340/500   0:00:11  loss=    -113.683  error=     11914.2
  345/500   0:00:11  loss=    -113.794  error=     11917.7
  350/500   0:00:11  loss=    -113.928  error=     11921.3
  355/500   0:00:12  loss=    -114.479  error=       11925
  360/500   0:00:12  loss=    -115.376  error=     11928.7
  365/500   0:00:12  loss=    -116.437  error=     11932.6
  370/500   0:00:12  loss=    -117.527  error=     11936.4
  375/500   0:00:12  loss=    -119.276  error=       11940
  380/500   0:00:12  loss=    -121.939  error=     11943.8
  385/500   0:00:13  loss=    -124.404  error=     11947.7
  390/500   0:00:13  loss=    -126.177  error=     11951.7
  395/500   0:00:13  loss=    -127.404  error=     11955.6
  400/500   0:00:13  loss=    -128.313  error=     11959.6
  405/500   0:00:13  loss=     -128.99  error=     11963.5
  410/500   0:00:13  loss=    -129.486  error=     11967.5
  415/500   0:00:14  loss=    -129.815  error=     11971.3
  420/500   0:00:14  loss=    -130.076  error=     11975.1
  425/500   0:00:14  loss=    -130.235  error=     11978.9
  430/500   0:00:14  loss=    -130.358  error=     11982.6
  435/500   0:00:14  loss=     -130.45  error=     11986.2
  440/500   0:00:14  loss=    -130.533  error=     11989.7
  445/500   0:00:15  loss=    -130.609  error=     11993.1
  450/500   0:00:15  loss=    -130.684  error=     11996.5
  455/500   0:00:15  loss=    -130.756  error=     11999.7
  460/500   0:00:15  loss=    -130.826  error=     12002.9
  465/500   0:00:15  loss=    -130.893  error=     12005.9
  470/500   0:00:15  loss=    -130.958  error=     12008.9
  475/500   0:00:16  loss=    -131.022  error=     12011.7
  480/500   0:00:16  loss=    -131.085  error=     12014.5
  485/500   0:00:16  loss=    -131.147  error=     12017.2
  490/500   0:00:16  loss=     -131.21  error=     12019.7
  495/500   0:00:16  loss=    -131.274  error=     12022.2
  500/500   0:00:17  loss=    -131.339  error=     12024.6
Finished

Optimization finished in 17.074 seconds
‣ Iterations: 500
‣ Final loss: -131.339
‣ Final error: 12024.6
==================================================

Trial 2 of 5

Starting optimization using Adam
‣ Model: SM
‣ Channels: 4
‣ Parameters: 40
‣ Training points: 381
‣ Initial loss: 128.845
‣ Initial error: 9718.72

Start Adam:
    0/500   0:00:00  loss=     128.845  error=     9718.72
    5/500   0:00:00  loss=     112.605  error=     10268.3
   10/500   0:00:00  loss=     97.8763  error=     10627.2
   15/500   0:00:00  loss=     84.5813  error=     10015.8
   20/500   0:00:00  loss=     71.9043  error=     9842.44
   25/500   0:00:00  loss=      59.449  error=     9901.86
   30/500   0:00:01  loss=     47.7542  error=      9700.2
   35/500   0:00:01  loss=     36.6144  error=     9654.51
   40/500   0:00:01  loss=     26.2795  error=     9628.91
   45/500   0:00:01  loss=     16.6807  error=     9523.71
   50/500   0:00:01  loss=     7.87683  error=     9501.54
   55/500   0:00:02  loss=  -0.0912034  error=     9439.07
   60/500   0:00:02  loss=    -7.23042  error=     9391.76
   65/500   0:00:02  loss=    -13.5651  error=      9355.5
   70/500   0:00:02  loss=    -19.1012  error=     9311.05
   75/500   0:00:02  loss=    -23.8963  error=     9274.01
   80/500   0:00:02  loss=    -27.9917  error=     9239.61
   85/500   0:00:03  loss=    -31.4435  error=     9206.57
   90/500   0:00:03  loss=    -34.3059  error=     9170.95
   95/500   0:00:03  loss=    -36.7176  error=     9141.99
  100/500   0:00:03  loss=    -38.6659  error=     9097.55
  105/500   0:00:03  loss=    -40.2447  error=     9031.76
  110/500   0:00:03  loss=    -41.8952  error=     8759.57
  115/500   0:00:04  loss=    -43.2706  error=     9014.15
  120/500   0:00:04  loss=    -44.3693  error=     8913.15
  125/500   0:00:04  loss=    -45.2192  error=      8725.1
  130/500   0:00:04  loss=    -45.8816  error=     8714.92
  135/500   0:00:04  loss=    -46.4923  error=     8792.61
  140/500   0:00:04  loss=    -46.9895  error=     8662.53
  145/500   0:00:05  loss=    -47.3833  error=     8634.11
  150/500   0:00:05  loss=    -47.6501  error=     8622.06
  155/500   0:00:05  loss=    -47.9179  error=     8577.89
  160/500   0:00:05  loss=    -48.2351  error=     8524.05
  165/500   0:00:05  loss=    -48.4893  error=     8467.46
  170/500   0:00:06  loss=    -48.6991  error=     8427.77
  175/500   0:00:06  loss=    -48.9169  error=        8365
  180/500   0:00:06  loss=    -49.1128  error=     8315.51
  185/500   0:00:06  loss=     -49.308  error=     8268.65
  190/500   0:00:06  loss=    -49.4565  error=     8220.97
  195/500   0:00:06  loss=    -49.6145  error=      8182.4
  200/500   0:00:07  loss=    -49.7395  error=     8145.97
  205/500   0:00:07  loss=    -49.8372  error=     8115.38
  210/500   0:00:07  loss=    -49.9703  error=     8090.06
  215/500   0:00:07  loss=    -50.0473  error=     8068.69
  220/500   0:00:07  loss=    -50.1443  error=     8050.94
  225/500   0:00:08  loss=    -50.2278  error=     8037.08
  230/500   0:00:08  loss=    -50.3144  error=      8026.3
  235/500   0:00:08  loss=    -50.4006  error=     8018.06
  240/500   0:00:08  loss=    -50.4715  error=     8011.85
  245/500   0:00:08  loss=     -50.538  error=     8007.34
  250/500   0:00:09  loss=    -50.5373  error=     8004.13
  255/500   0:00:09  loss=     -50.671  error=     8002.16
  260/500   0:00:09  loss=    -50.7322  error=     8001.24
  265/500   0:00:09  loss=    -50.7664  error=     8000.59
  270/500   0:00:09  loss=    -50.8382  error=     8001.11
  275/500   0:00:10  loss=    -50.8966  error=     8001.17
  280/500   0:00:10  loss=     -50.949  error=     8001.96
  285/500   0:00:10  loss=    -51.0208  error=     8003.21
  290/500   0:00:10  loss=    -51.0758  error=     8004.03
  295/500   0:00:10  loss=    -51.1342  error=      8005.3
  300/500   0:00:11  loss=    -51.1876  error=     8006.59
  305/500   0:00:11  loss=    -51.2299  error=     8007.75
  310/500   0:00:11  loss=    -51.2553  error=     8009.84
  315/500   0:00:11  loss=    -51.3278  error=     8010.75
  320/500   0:00:11  loss=    -51.3754  error=     8011.93
  325/500   0:00:11  loss=     -51.412  error=     8013.29
  330/500   0:00:12  loss=    -51.4445  error=     8014.65
  335/500   0:00:12  loss=    -51.4167  error=     8015.91
  340/500   0:00:12  loss=    -51.5247  error=     8017.06
  345/500   0:00:12  loss=    -51.5537  error=     8017.85
  350/500   0:00:12  loss=      -51.58  error=     8018.26
  355/500   0:00:12  loss=    -51.6088  error=     8018.46
  360/500   0:00:13  loss=    -51.6372  error=     8018.31
  365/500   0:00:13  loss=    -51.6646  error=     8017.62
  370/500   0:00:13  loss=    -51.6806  error=     8017.01
  375/500   0:00:13  loss=    -51.6934  error=      8015.3
  380/500   0:00:13  loss=    -51.7323  error=      8013.5
  385/500   0:00:13  loss=    -51.7559  error=     8011.56
  390/500   0:00:14  loss=     -51.739  error=     8009.12
  395/500   0:00:14  loss=    -51.7972  error=     8006.08
  400/500   0:00:14  loss=    -51.8137  error=     8002.64
  405/500   0:00:14  loss=    -51.8202  error=      7998.8
  410/500   0:00:14  loss=    -51.8357  error=     7994.36
  415/500   0:00:15  loss=       -51.8  error=      7988.8
  420/500   0:00:15  loss=    -51.8914  error=     7981.95
  425/500   0:00:15  loss=    -51.8937  error=      7972.7
  430/500   0:00:15  loss=    -51.9109  error=     7958.54
  435/500   0:00:15  loss=    -51.9203  error=     7931.96
  440/500   0:00:15  loss=     -51.974  error=     7826.42
  445/500   0:00:16  loss=    -57.9888  error=     8867.18
  450/500   0:00:16  loss=    -57.0901  error=     8006.72
  455/500   0:00:16  loss=    -59.7622  error=     8136.61
  460/500   0:00:16  loss=    -60.8646  error=     8157.89
  465/500   0:00:16  loss=    -62.2998  error=      7164.7
  470/500   0:00:17  loss=    -62.9866  error=     7670.34
  475/500   0:00:17  loss=    -63.9367  error=     7263.65
  480/500   0:00:17  loss=    -64.8784  error=     6950.86
  485/500   0:00:17  loss=    -65.6348  error=      6492.4
  490/500   0:00:17  loss=    -65.9036  error=     6271.15
  495/500   0:00:17  loss=    -66.3389  error=     6441.74
  500/500   0:00:18  loss=    -66.5022  error=     6419.53
Finished

Optimization finished in 18.102 seconds
‣ Iterations: 500
‣ Final loss: -66.5022
‣ Final error: 6419.53
==================================================

Trial 3 of 5

Starting optimization using Adam
‣ Model: SM
‣ Channels: 4
‣ Parameters: 40
‣ Training points: 383
‣ Initial loss: 126.644
‣ Initial error: 12734.7

Start Adam:
    0/500   0:00:00  loss=     126.644  error=     12734.7
    5/500   0:00:00  loss=     107.883  error=     12323.9
   10/500   0:00:00  loss=     92.8432  error=       12625
   15/500   0:00:00  loss=     78.0226  error=     12173.9
   20/500   0:00:00  loss=     63.7247  error=     12153.6
   25/500   0:00:00  loss=      49.786  error=     12157.5
   30/500   0:00:01  loss=     36.2407  error=     12083.8
   35/500   0:00:01  loss=     23.1753  error=       12065
   40/500   0:00:01  loss=     10.6744  error=     12080.3
   45/500   0:00:01  loss=    -1.27614  error=     11993.5
   50/500   0:00:01  loss=    -12.6352  error=     12030.5
   55/500   0:00:01  loss=    -23.3468  error=     12007.1
   60/500   0:00:02  loss=    -33.4051  error=     11967.5
   65/500   0:00:02  loss=    -42.7741  error=     11954.7
   70/500   0:00:02  loss=     -51.432  error=     11948.8
   75/500   0:00:02  loss=    -59.3783  error=     11936.2
   80/500   0:00:02  loss=    -66.6121  error=     11916.1
   85/500   0:00:02  loss=    -73.1444  error=     11896.6
   90/500   0:00:03  loss=    -78.9864  error=     11880.8
   95/500   0:00:03  loss=    -84.1546  error=     11863.7
  100/500   0:00:03  loss=    -88.7264  error=       11848
  105/500   0:00:03  loss=    -92.6821  error=     11832.3
  110/500   0:00:03  loss=    -96.0797  error=     11817.7
  115/500   0:00:04  loss=    -98.9338  error=     11802.4
  120/500   0:00:04  loss=    -101.342  error=     11789.8
  125/500   0:00:04  loss=    -103.299  error=     11777.7
  130/500   0:00:04  loss=    -104.896  error=     11768.4
  135/500   0:00:04  loss=    -106.188  error=     11756.8
  140/500   0:00:04  loss=    -107.261  error=     11750.9
  145/500   0:00:05  loss=     -108.27  error=     11742.5
  150/500   0:00:05  loss=    -109.232  error=     11738.4
  155/500   0:00:05  loss=    -110.091  error=     11731.8
  160/500   0:00:05  loss=    -110.844  error=     11729.3
  165/500   0:00:05  loss=    -111.522  error=     11725.4
  170/500   0:00:05  loss=    -112.906  error=     11722.4
  175/500   0:00:06  loss=    -114.972  error=     11721.1
  180/500   0:00:06  loss=    -116.959  error=       11720
  185/500   0:00:06  loss=    -118.987  error=     11719.6
  190/500   0:00:06  loss=    -120.965  error=     11719.2
  195/500   0:00:06  loss=    -122.874  error=     11719.1
  200/500   0:00:06  loss=    -124.804  error=     11720.1
  205/500   0:00:07  loss=    -126.731  error=       11722
  210/500   0:00:07  loss=    -128.273  error=     11719.8
  215/500   0:00:07  loss=    -129.582  error=     11722.9
  220/500   0:00:07  loss=    -130.646  error=     11728.9
  225/500   0:00:07  loss=    -131.455  error=       11735
  230/500   0:00:07  loss=    -132.155  error=       11738
  235/500   0:00:08  loss=     -132.72  error=     11741.9
  240/500   0:00:08  loss=      -133.2  error=     11747.9
  245/500   0:00:08  loss=    -133.608  error=     11755.4
  250/500   0:00:08  loss=    -133.955  error=     11762.8
  255/500   0:00:08  loss=    -134.261  error=     11770.4
  260/500   0:00:08  loss=    -134.537  error=     11778.1
  265/500   0:00:09  loss=    -134.791  error=     11786.2
  270/500   0:00:09  loss=    -135.027  error=     11794.4
  275/500   0:00:09  loss=    -135.248  error=     11802.6
  280/500   0:00:09  loss=    -135.453  error=     11810.9
  285/500   0:00:09  loss=    -135.629  error=     11819.8
  290/500   0:00:09  loss=    -135.823  error=       11828
  295/500   0:00:10  loss=    -135.989  error=     11836.3
  300/500   0:00:10  loss=    -136.149  error=     11844.3
  305/500   0:00:10  loss=    -136.297  error=     11851.9
  310/500   0:00:10  loss=    -136.415  error=     11858.8
  315/500   0:00:10  loss=    -136.519  error=     11841.1
  320/500   0:00:11  loss=    -136.541  error=     11840.1
  325/500   0:00:11  loss=    -136.741  error=     11854.8
  330/500   0:00:11  loss=    -136.864  error=     11916.6
  335/500   0:00:11  loss=    -136.962  error=     11857.1
  340/500   0:00:11  loss=    -137.053  error=     11906.8
  345/500   0:00:11  loss=    -137.134  error=     11895.1
  350/500   0:00:12  loss=     -137.21  error=       11891
  355/500   0:00:12  loss=    -137.284  error=     11909.9
  360/500   0:00:12  loss=     -137.35  error=       11895
  365/500   0:00:12  loss=    -137.412  error=     11905.7
  370/500   0:00:12  loss=    -137.471  error=     11909.1
  375/500   0:00:12  loss=    -137.525  error=     11902.9
  380/500   0:00:13  loss=    -137.559  error=     11908.4
  385/500   0:00:13  loss=    -137.624  error=     11911.4
  390/500   0:00:13  loss=    -137.673  error=       11909
  395/500   0:00:13  loss=    -137.732  error=       11910
  400/500   0:00:13  loss=     -137.79  error=     11913.1
  405/500   0:00:13  loss=    -137.846  error=     11914.4
  410/500   0:00:14  loss=      -137.9  error=     11914.8
  415/500   0:00:14  loss=    -137.955  error=       11916
  420/500   0:00:14  loss=    -138.013  error=     11918.1
  425/500   0:00:14  loss=    -138.075  error=     11920.3
  430/500   0:00:14  loss=     -138.14  error=     11922.4
  435/500   0:00:14  loss=    -138.211  error=     11924.6
  440/500   0:00:15  loss=     -138.29  error=     11926.8
  445/500   0:00:15  loss=    -138.379  error=     11929.2
  450/500   0:00:15  loss=    -138.483  error=     11931.7
  455/500   0:00:15  loss=    -138.611  error=     11934.3
  460/500   0:00:15  loss=    -138.773  error=       11937
  465/500   0:00:15  loss=    -138.988  error=     11939.7
  470/500   0:00:16  loss=    -139.265  error=     11942.5
  475/500   0:00:16  loss=    -139.605  error=     11945.1
  480/500   0:00:16  loss=    -140.082  error=     11947.8
  485/500   0:00:16  loss=    -140.621  error=     11950.3
  490/500   0:00:16  loss=    -141.262  error=     11952.8
  495/500   0:00:16  loss=    -142.072  error=     11955.2
  500/500   0:00:17  loss=    -143.113  error=     11957.5
Finished

Optimization finished in 17.151 seconds
‣ Iterations: 500
‣ Final loss: -143.113
‣ Final error: 11957.5
==================================================

Trial 4 of 5

Starting optimization using Adam
‣ Model: SM
‣ Channels: 4
‣ Parameters: 40
‣ Training points: 380
‣ Initial loss: 120.852
‣ Initial error: 9901.25

Start Adam:
    0/500   0:00:00  loss=     120.852  error=     9901.25
    5/500   0:00:00  loss=     101.031  error=     11065.7
   10/500   0:00:00  loss=     86.8717  error=     12768.9
   15/500   0:00:00  loss=     73.4981  error=     12995.1
   20/500   0:00:00  loss=     59.6219  error=     10965.8
   25/500   0:00:00  loss=     47.0914  error=     10829.3
   30/500   0:00:01  loss=     35.3683  error=     11142.5
   35/500   0:00:01  loss=     23.9623  error=     10704.4
   40/500   0:00:01  loss=     13.4593  error=     10786.1
   45/500   0:00:01  loss=     3.71044  error=     10712.1
   50/500   0:00:01  loss=     -5.2067  error=     10688.7
   55/500   0:00:01  loss=    -13.3355  error=       10634
   60/500   0:00:02  loss=    -20.6906  error=     10630.5
   65/500   0:00:02  loss=    -27.2093  error=     10591.4
   70/500   0:00:02  loss=    -32.9291  error=     10575.8
   75/500   0:00:02  loss=    -37.8846  error=     10537.5
   80/500   0:00:02  loss=    -42.2284  error=     10523.8
   85/500   0:00:02  loss=    -46.6129  error=     10502.5
   90/500   0:00:03  loss=    -51.1645  error=     10476.4
   95/500   0:00:03  loss=    -54.5717  error=     10459.1
  100/500   0:00:03  loss=    -56.9409  error=     10443.6
  105/500   0:00:03  loss=    -59.1152  error=     10428.9
  110/500   0:00:03  loss=    -60.7783  error=     10415.4
  115/500   0:00:03  loss=     -62.182  error=     10402.9
  120/500   0:00:04  loss=    -63.2461  error=     10392.4
  125/500   0:00:04  loss=    -64.1148  error=     10384.3
  130/500   0:00:04  loss=    -64.7284  error=     10380.2
  135/500   0:00:04  loss=    -65.2666  error=     10374.3
  140/500   0:00:04  loss=    -65.6969  error=     10369.9
  145/500   0:00:04  loss=    -66.0378  error=     10366.3
  150/500   0:00:05  loss=    -66.3065  error=     10363.9
  155/500   0:00:05  loss=    -66.5305  error=     10363.2
  160/500   0:00:05  loss=    -66.7221  error=     10363.7
  165/500   0:00:05  loss=    -66.8903  error=       10365
  170/500   0:00:05  loss=     -67.039  error=     10367.4
  175/500   0:00:05  loss=      -67.17  error=     10370.7
  180/500   0:00:06  loss=    -67.2921  error=     10373.7
  185/500   0:00:06  loss=    -67.4031  error=     10378.2
  190/500   0:00:06  loss=    -67.5019  error=     10383.6
  195/500   0:00:06  loss=    -67.5752  error=     10387.9
  200/500   0:00:06  loss=    -67.6831  error=     10395.4
  205/500   0:00:06  loss=    -67.7164  error=     10400.6
  210/500   0:00:06  loss=    -67.8432  error=     10411.1
  215/500   0:00:07  loss=    -67.9192  error=     10419.3
  220/500   0:00:07  loss=    -67.9838  error=     10426.3
  225/500   0:00:07  loss=    -68.0617  error=     10437.2
  230/500   0:00:07  loss=    -68.1255  error=     10447.4
  235/500   0:00:07  loss=    -68.1902  error=     10457.2
  240/500   0:00:07  loss=    -68.2536  error=     10468.5
  245/500   0:00:08  loss=    -68.3155  error=     10480.7
  250/500   0:00:08  loss=    -68.3769  error=     10492.7
  255/500   0:00:08  loss=    -68.4387  error=     10504.5
  260/500   0:00:08  loss=    -68.4995  error=     10516.1
  265/500   0:00:08  loss=    -68.5594  error=     10527.1
  270/500   0:00:08  loss=    -68.6177  error=     10537.2
  275/500   0:00:09  loss=    -68.6733  error=       10546
  280/500   0:00:09  loss=    -68.7259  error=     10553.6
  285/500   0:00:09  loss=    -68.7754  error=     10559.9
  290/500   0:00:09  loss=    -68.8219  error=     10565.1
  295/500   0:00:09  loss=    -68.8657  error=     10569.3
  300/500   0:00:09  loss=    -68.9071  error=     10572.7
  305/500   0:00:10  loss=    -68.9466  error=     10575.5
  310/500   0:00:10  loss=    -68.9843  error=     10577.7
  315/500   0:00:10  loss=    -69.0199  error=     10579.6
  320/500   0:00:10  loss=    -69.0404  error=     10581.4
  325/500   0:00:10  loss=    -69.0809  error=     10582.9
  330/500   0:00:10  loss=    -69.1172  error=     10584.9
  335/500   0:00:11  loss=    -69.1195  error=     10581.8
  340/500   0:00:11  loss=    -69.1076  error=     10634.2
  345/500   0:00:11  loss=    -69.1914  error=     10612.6
  350/500   0:00:11  loss=    -69.1957  error=     10562.2
  355/500   0:00:11  loss=    -69.2389  error=       10607
  360/500   0:00:12  loss=    -69.2866  error=       10579
  365/500   0:00:12  loss=    -69.2875  error=     10599.9
  370/500   0:00:12  loss=    -69.3171  error=     10593.2
  375/500   0:00:12  loss=    -69.2916  error=     10593.8
  380/500   0:00:12  loss=    -69.3479  error=     10601.1
  385/500   0:00:12  loss=     -69.405  error=     10592.9
  390/500   0:00:13  loss=    -69.4093  error=     10601.6
  395/500   0:00:13  loss=    -69.4248  error=     10599.4
  400/500   0:00:13  loss=    -69.4607  error=       10599
  405/500   0:00:13  loss=    -69.4835  error=     10603.4
  410/500   0:00:13  loss=    -69.4983  error=     10602.4
  415/500   0:00:13  loss=    -69.5191  error=     10602.6
  420/500   0:00:14  loss=    -69.5359  error=     10605.1
  425/500   0:00:14  loss=    -69.5532  error=     10605.7
  430/500   0:00:14  loss=    -69.5691  error=     10605.7
  435/500   0:00:14  loss=    -69.5852  error=     10606.7
  440/500   0:00:14  loss=    -69.6003  error=     10607.9
  445/500   0:00:14  loss=     -69.615  error=     10608.6
  450/500   0:00:15  loss=    -69.6294  error=       10609
  455/500   0:00:15  loss=    -69.6434  error=     10609.6
  460/500   0:00:15  loss=    -69.6567  error=     10610.3
  465/500   0:00:15  loss=    -69.6631  error=     10611.1
  470/500   0:00:15  loss=    -69.6825  error=     10611.8
  475/500   0:00:15  loss=    -69.6941  error=     10612.6
  480/500   0:00:16  loss=    -69.7019  error=     10612.3
  485/500   0:00:16  loss=    -69.6335  error=     10617.7
  490/500   0:00:16  loss=    -69.7239  error=     10615.2
  495/500   0:00:16  loss=    -69.7363  error=     10615.6
  500/500   0:00:16  loss=    -69.6438  error=     10616.4
Finished

Optimization finished in 16.784 seconds
‣ Iterations: 500
‣ Final loss: -69.6438
‣ Final error: 10616.4
==================================================

Trial 5 of 5

Starting optimization using Adam
‣ Model: SM
‣ Channels: 4
‣ Parameters: 40
‣ Training points: 387
‣ Initial loss: 136.686
‣ Initial error:  12397

Start Adam:
    0/500   0:00:00  loss=     136.686  error=       12397
    5/500   0:00:00  loss=     118.681  error=     12200.9
   10/500   0:00:00  loss=      103.93  error=     12245.9
   15/500   0:00:00  loss=     89.8962  error=       13637
   20/500   0:00:00  loss=     76.2511  error=     12737.3
   25/500   0:00:00  loss=     63.1651  error=       13432
   30/500   0:00:01  loss=     50.5653  error=     13050.1
   35/500   0:00:01  loss=     38.5746  error=     13409.2
   40/500   0:00:01  loss=       27.17  error=     13066.4
   45/500   0:00:01  loss=     15.4147  error=     13306.2
   50/500   0:00:01  loss=     5.13405  error=     13111.6
   55/500   0:00:01  loss=    -4.37322  error=     12895.5
   60/500   0:00:02  loss=     -13.425  error=     11833.2
   65/500   0:00:02  loss=    -21.5903  error=     11512.1
   70/500   0:00:02  loss=    -28.9507  error=       11783
   75/500   0:00:02  loss=    -35.4939  error=     11357.7
   80/500   0:00:02  loss=    -41.2221  error=     11436.5
   85/500   0:00:02  loss=    -46.1689  error=     11280.9
   90/500   0:00:03  loss=     -50.389  error=     11226.3
   95/500   0:00:03  loss=    -53.8908  error=     11140.5
  100/500   0:00:03  loss=    -56.8579  error=     11084.5
  105/500   0:00:03  loss=    -59.2751  error=       11016
  110/500   0:00:03  loss=    -61.2387  error=     10968.1
  115/500   0:00:04  loss=    -62.8149  error=     10905.8
  120/500   0:00:04  loss=    -64.0748  error=       10868
  125/500   0:00:04  loss=    -65.0507  error=     10811.1
  130/500   0:00:04  loss=    -65.8394  error=     10777.7
  135/500   0:00:04  loss=    -66.4168  error=     10734.8
  140/500   0:00:04  loss=    -66.8993  error=     10700.8
  145/500   0:00:05  loss=    -67.2602  error=     10670.1
  150/500   0:00:05  loss=    -67.5197  error=     10641.4
  155/500   0:00:05  loss=    -67.7421  error=     10619.2
  160/500   0:00:05  loss=    -67.9046  error=     10597.1
  165/500   0:00:05  loss=    -68.0314  error=     10579.1
  170/500   0:00:05  loss=    -68.1288  error=     10563.7
  175/500   0:00:06  loss=    -68.2118  error=     10549.8
  180/500   0:00:06  loss=    -68.2777  error=     10539.9
  185/500   0:00:06  loss=    -68.3399  error=     10529.1
  190/500   0:00:06  loss=    -68.3955  error=     10521.6
  195/500   0:00:06  loss=    -68.4223  error=     10512.8
  200/500   0:00:07  loss=    -68.4932  error=     10508.7
  205/500   0:00:07  loss=    -68.5377  error=     10503.7
  210/500   0:00:07  loss=     -68.617  error=       10499
  215/500   0:00:07  loss=    -69.3075  error=     10494.6
  220/500   0:00:07  loss=    -70.0798  error=     10490.3
  225/500   0:00:07  loss=    -71.4119  error=     10486.4
  230/500   0:00:08  loss=    -73.4564  error=     10483.6
  235/500   0:00:08  loss=    -75.9524  error=     10481.7
  240/500   0:00:08  loss=    -78.3254  error=       10478
  245/500   0:00:08  loss=    -80.2907  error=     10477.2
  250/500   0:00:08  loss=    -81.8411  error=     10472.8
  255/500   0:00:09  loss=    -83.1045  error=     10472.3
  260/500   0:00:09  loss=    -84.0776  error=     10471.3
  265/500   0:00:09  loss=    -84.8676  error=     10468.2
  270/500   0:00:09  loss=    -85.4743  error=     10466.6
  275/500   0:00:09  loss=    -85.9282  error=     10466.5
  280/500   0:00:09  loss=    -86.2717  error=     10463.2
  285/500   0:00:10  loss=    -86.5368  error=     10463.3
  290/500   0:00:10  loss=    -86.7129  error=     10461.7
  295/500   0:00:10  loss=    -86.8369  error=     10460.2
  300/500   0:00:10  loss=    -86.8992  error=     10460.9
  305/500   0:00:10  loss=    -86.9293  error=     10456.5
  310/500   0:00:10  loss=    -86.9919  error=     10455.9
  315/500   0:00:11  loss=    -87.0431  error=     10455.4
  320/500   0:00:11  loss=    -87.0819  error=     10454.9
  325/500   0:00:11  loss=    -87.1104  error=     10454.3
  330/500   0:00:11  loss=    -87.1205  error=     10453.9
  335/500   0:00:11  loss=    -87.1643  error=     10453.5
  340/500   0:00:12  loss=    -87.1911  error=     10453.2
  345/500   0:00:12  loss=    -87.2095  error=     10452.8
  350/500   0:00:12  loss=    -87.2282  error=     10452.2
  355/500   0:00:12  loss=     -87.247  error=     10451.3
  360/500   0:00:12  loss=    -87.2653  error=     10450.8
  365/500   0:00:12  loss=    -87.2821  error=     10450.7
  370/500   0:00:13  loss=    -87.2976  error=     10449.8
  375/500   0:00:13  loss=    -87.3113  error=     10450.1
  380/500   0:00:13  loss=    -87.3065  error=     10447.9
  385/500   0:00:13  loss=    -87.2777  error=     10451.5
  390/500   0:00:13  loss=    -87.3281  error=     10450.3
  395/500   0:00:13  loss=    -87.3643  error=     10449.1
  400/500   0:00:14  loss=    -87.3846  error=     10448.2
  405/500   0:00:14  loss=     -87.398  error=     10447.5
  410/500   0:00:14  loss=    -87.4093  error=     10446.9
  415/500   0:00:14  loss=    -87.4228  error=     10446.7
  420/500   0:00:14  loss=    -87.4391  error=     10446.7
  425/500   0:00:15  loss=    -87.4534  error=     10446.9
  430/500   0:00:15  loss=    -87.4619  error=     10446.9
  435/500   0:00:15  loss=    -87.4623  error=     10446.4
  440/500   0:00:15  loss=     -87.496  error=       10446
  445/500   0:00:15  loss=    -87.5137  error=     10446.3
  450/500   0:00:15  loss=    -87.5283  error=     10445.8
  455/500   0:00:16  loss=    -87.5464  error=       10446
  460/500   0:00:16  loss=    -87.5637  error=     10445.1
  465/500   0:00:16  loss=    -87.5476  error=     10447.6
  470/500   0:00:16  loss=     -87.552  error=     10443.3
  475/500   0:00:16  loss=    -87.5836  error=     10443.4
  480/500   0:00:16  loss=    -87.6276  error=     10443.7
  485/500   0:00:17  loss=    -87.6653  error=       10444
  490/500   0:00:17  loss=    -87.7002  error=     10444.2
  495/500   0:00:17  loss=    -87.7342  error=     10444.5
  500/500   0:00:17  loss=    -87.7683  error=     10444.7
Finished

Optimization finished in 17.678 seconds
‣ Iterations: 500
‣ Final loss: -87.7683
‣ Final error: 10444.7
==================================================
In [15]:
pd.DataFrame(np.c_[sm_mae.mean(1), sm_rmse.mean(1), sm_mape.mean(1),
             sm_mae.std(1), sm_rmse.std(1), sm_mape.std(1)],
             columns=['MAE', 'RMSE', 'MAPE', 'MAE std', 'RMSE std', 'MAPE std'])
Out[15]:
MAE RMSE MAPE MAE std RMSE std MAPE std
0 42.899739 60.453239 2.127166 59.796056 85.269113 1.133937
1 34.414545 46.843570 3.104896 43.693042 60.431288 2.640380
2 43.929931 60.689220 2.278018 59.940593 84.262763 0.873423
3 41.556311 58.398776 3.545758 55.198414 79.120772 3.148026
4 43.104613 57.201560 2.829461 58.456193 77.638660 2.093766
In [16]:
best_sm = sm_models[np.argmin(sm_mape.mean(1))]
best_sm.plot_prediction();

Results¶

In [17]:
print('MAPE errors per model')
print('MOSM:    %g ± %g' % (mosm_mape.mean(1).mean(), mosm_mape.mean(1).std()))
print('CSM:     %g ± %g' % (csm_mape.mean(1).mean(), csm_mape.mean(1).std()))
print('SM-LMC:  %g ± %g' % (smlmc_mape.mean(1).mean(), smlmc_mape.mean(1).std()))
print('SM:      %g ± %g' % (sm_mape.mean(1).mean(), sm_mape.mean(1).std()))
MAPE errors per model
MOSM:    3.10081 ± 0.0291348
CSM:     3.52736 ± 0.183989
SM-LMC:  2.91661 ± 0.327235
SM:      2.77706 ± 0.523932