mbtr package

Submodules

mbtr.losses module

class mbtr.losses.FourierLoss(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

Loss for the Fourier regression:

\[\mathcal{L} = \Vert y - P x\Vert_2^2\]

where \(P\) is the projection matrix:

\[P=\left[\left[\cos \left(k \frac{2 \pi t}{n_{t}}\right)^{T}, \sin \left(k \frac{2 \pi t}{n_{t}}\right)^{T}\right]^{T}\right]_{k \in \mathcal{K}}\]
eval_optimal_loss(G2, H)

Evaluate the optimal loss (using response obtained minimizing the second order loss approximation).

Parameters
  • G2 – squared sum of gradients in the current leaf

  • H – sum of Hessians diags in the current leaf

Returns

optimal loss, scalar

eval_optimal_response(G, H)

Evaluate optimal response, given G and H.

Parameters
  • G – mean gradient for the current leaf.

  • H – mean Hessian for the current leaf.

Returns

optimal response under second order loss approximation.

get_initial_guess(y)

Return an initial guess for the prediciton. This can be loss-specific.

Parameters

y – target matrix of the training set

Returns

np.ndarray with initial guess

projection_matrix(n)

Return projection matrix for the Fourier coefficient estimation.

Parameters

n – number of observations

Returns

projection matrix P, (2*n_harmonics, n_t) where n_harmonics is the number of harmonics to fit, n_t the

target dimension

required_pars = ['n_harmonics']
set_dimension(n_dim)

Initialize all the properties which depends on the dimension of the target

Parameters

n_dims – dimension of the target

Returns

None

class mbtr.losses.LatentVariable(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

Loss for the hierarchical reconciliation problem, in the form:

\[\mathcal{L} = \Vert y - S x\Vert_2^2\]

where \(S\) is the hierarchy matrix. The initial guess is the mean of the last columns of y.

compute_H_inv(n)
compute_fast_H_hat(n)
eval_optimal_loss(yy, H)

Evaluate the optimal loss (using response obtained minimizing the second order loss approximation).

Parameters
  • G2 – squared sum of gradients in the current leaf

  • H – sum of Hessians diags in the current leaf

Returns

optimal loss, scalar

eval_optimal_response(G, H)

Evaluate optimal response, given G and H.

Parameters
  • G – mean gradient for the current leaf.

  • H – mean Hessian for the current leaf.

Returns

optimal response under second order loss approximation.

get_grad_and_hessian_diags(y, y_hat, iteration, leaves_idx)

Return the loss gradient and loss Hessian’s diagonals based on the current model estimation y_hat and target y matrices. Instead of returning the full Hessian (a 3rd order tensor), the method returns only the Hessian diagonals for each observation, stored in a (n_obs, n_t) matrix. These diagonals are then used by the loss to reconstruct the full Hessian with appropriate dimensions and structure. Currently, full Hessians inferred by data are not supported.

Parameters
  • y – target matrix (n_obs, n_t)

  • y_hat – current target estimation matrix (n_obs, n_t)

  • iteration – current number of iteration, generally not needed

  • leaves_idx – leaves’ indexes for each observation in y, (n_obs, 1). This is needed for example by mbtr.losses.QuadraticQuantileLoss.

Returns

grad, hessian_diags tuple, each of which is a (n_obs, n_t) matrix

get_initial_guess(y)

The initial guess is generated from the last columns of the target matrix, as:

\[y_0 = \left( \mathbb{E} y_b \right) S^T\]

where \(\mathbb{E}\) is the expectation (row-mean), \(S\) is the hierarchy matrix, and \(y_b\) stands for the last columns of y, with dimension (n_obs, n_b), where n_b is the number of bottom series.

Parameters

y – target matrix of the training set

Returns

np.ndarray with initial guess

required_pars = ['S', 'precision']
set_dimension(n_dims)

For the latent loss the number of dimensions is equal to the second dimension of the S matrix, and must not be inferred from the target

class mbtr.losses.LinRegLoss(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

eval_optimal_loss(G, x)

Evaluate the optimal loss (using response obtained minimizing the second order loss approximation).

Parameters
  • G – gradient for the current leaf.

  • x – linear regression features for the current leaf.

Returns

optimal loss, scalar

eval_optimal_response(G, x)

Evaluate optimal response, given G and x. This is done computing a Ridge regression with intercept

\[w = \left(\tilde{x}^T \tilde{x} + \lambda I \right)^{-1} \left(\tilde{x}^T G \right)\]

where \(\tilde{x}\) is the \(x\) matrix augmented with an unitary column and \(\lambda\) is the Ridge coefficient.

Parameters
  • G – gradient for the current leaf.

  • x – linear regression features for the current leaf.

Returns

optimal response under second order loss approximation.

set_dimension(n_dim)

Initialize all the properties which depends on the dimension of the target

Parameters

n_dims – dimension of the target

Returns

None

class mbtr.losses.Loss(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: object

Loss function class. A loss is defined by its gradient and Hessians. Note that if your specific loss funciton requires some additional argument, you can specify it in the required_pars. Upon instantiation, this list will be used to check if loss_kwargs contains all the needed parameters. Each class inheriting from mbtr.losses.Loss must provide an H_inv method, computing the inverse of the Hessian.

Parameters
  • lambda_weights – quadratic penalization parameter for the leaves weights

  • lambda_leaves – quadratic penalization parameter for the number of leaves

  • loss_kwargs – additional parameters needed for a specific loss type

H_inv(H)

Computes the inverse of the Hessian, given the Hessian’s diagonal of the current leave. The default implements MSE inverse.

Parameters

H – current leaf Hessian’s diagonal (n_t)

Returns

inv(H), (n_t, n_t)

eval(y, y_hat, trees)

Evaluates the overall loss, which is composed by the tree’s loss plus weights and total leaves penalizations

Parameters
  • y – observations

  • y_hat – current :class: mbtr.MBT estimations

  • trees – array of fitted trees up to the current iteartion

Returns

tree loss and regularizations loss tuple, scalars

eval_optimal_loss(G2, H)

Evaluate the optimal loss (using response obtained minimizing the second order loss approximation).

Parameters
  • G2 – squared sum of gradients in the current leaf

  • H – sum of Hessians diags in the current leaf

Returns

optimal loss, scalar

eval_optimal_response(G, H)

Evaluate optimal response, given G and H.

Parameters
  • G – mean gradient for the current leaf.

  • H – mean Hessian for the current leaf.

Returns

optimal response under second order loss approximation.

get_grad_and_hessian_diags(y, y_hat, iteration, leaves_idx)

Return the loss gradient and loss Hessian’s diagonals based on the current model estimation y_hat and target y matrices. Instead of returning the full Hessian (a 3rd order tensor), the method returns only the Hessian diagonals for each observation, stored in a (n_obs, n_t) matrix. These diagonals are then used by the loss to reconstruct the full Hessian with appropriate dimensions and structure. Currently, full Hessians inferred by data are not supported.

Parameters
  • y – target matrix (n_obs, n_t)

  • y_hat – current target estimation matrix (n_obs, n_t)

  • iteration – current number of iteration, generally not needed

  • leaves_idx – leaves’ indexes for each observation in y, (n_obs, 1). This is needed for example by mbtr.losses.QuadraticQuantileLoss.

Returns

grad, hessian_diags tuple, each of which is a (n_obs, n_t) matrix

get_initial_guess(y)

Return an initial guess for the prediciton. This can be loss-specific.

Parameters

y – target matrix of the training set

Returns

np.ndarray with initial guess

required_pars = []
set_dimension(n_dims)

Initialize all the properties which depends on the dimension of the target

Parameters

n_dims – dimension of the target

Returns

None

tree_loss(y, y_hat)

Compute the tree loss (without penalizations)

Parameters
  • y – observations of the target on the traning set

  • y_hat – current estimation of the MBT

Returns

tree loss

class mbtr.losses.MSE(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

Mean Squared Error loss, a.k.a. L2, Ordinary Least Squares.

\[\mathcal{L} = \Vert y - w\Vert_2^2 + \frac{1}{2} w^T \Lambda w\]

where \(\Lambda\) is the quadratic punishment matrix.

Parameters
  • lambda_weights – quadratic penalization parameter for the leaves weights

  • lambda_leaves – quadratic penalization parameter for the number of leaves

  • loss_kwargs – additional parameters needed for a specific loss type

eval_optimal_loss(G2, H)

Evaluate the optimal loss (using response obtained minimizing the second order loss approximation).

Parameters
  • G2 – squared sum of gradients in the current leaf

  • H – sum of Hessians diags in the current leaf

Returns

optimal loss, scalar

eval_optimal_response(G, H)

Evaluate optimal response, given G and H.

Parameters
  • G – mean gradient for the current leaf.

  • H – mean Hessian for the current leaf.

Returns

optimal response under second order loss approximation.

set_dimension(n_dim)

Initialize all the properties which depends on the dimension of the target

Parameters

n_dims – dimension of the target

Returns

None

class mbtr.losses.QuadraticQuantileLoss(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

H_inv(H)

Computes the inverse of the Hessian, given the Hessian’s diagonal of the current leave. The default implements MSE inverse.

Parameters

H – current leaf Hessian’s diagonal (n_t)

Returns

inv(H), (n_t, n_t)

exact_response(y)
get_grad_and_hessian_diags(y, y_hat, iteration, leaves_idx)

Return the loss gradient and loss Hessian’s diagonals based on the current model estimation y_hat and target y matrices. Instead of returning the full Hessian (a 3rd order tensor), the method returns only the Hessian diagonals for each observation, stored in a (n_obs, n_t) matrix. These diagonals are then used by the loss to reconstruct the full Hessian with appropriate dimensions and structure. Currently, full Hessians inferred by data are not supported.

Parameters
  • y – target matrix (n_obs, n_t)

  • y_hat – current target estimation matrix (n_obs, n_t)

  • iteration – current number of iteration, generally not needed

  • leaves_idx – leaves’ indexes for each observation in y, (n_obs, 1). This is needed for example by mbtr.losses.QuadraticQuantileLoss.

Returns

grad, hessian_diags tuple, each of which is a (n_obs, n_t) matrix

get_initial_guess(y)

The initial guess are the alpha quantiles of the target matrix y.

Parameters

y – target matrix of the training set

Returns

np.ndarray with initial guess

required_pars = ['alphas']
tree_loss(y, y_hat)

Compute the tree loss (without penalizations)

Parameters
  • y – observations of the target on the traning set

  • y_hat – current estimation of the MBT

Returns

tree loss

class mbtr.losses.QuantileLoss(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

H_inv(H)

Computes the inverse of the Hessian, given the Hessian’s diagonal of the current leave. The default implements MSE inverse.

Parameters

H – current leaf Hessian’s diagonal (n_t)

Returns

inv(H), (n_t, n_t)

exact_response(y)
get_grad_and_hessian_diags(y, y_hat, iteration, leaves_idx)

Return the loss gradient and loss Hessian’s diagonals based on the current model estimation y_hat and target y matrices. Instead of returning the full Hessian (a 3rd order tensor), the method returns only the Hessian diagonals for each observation, stored in a (n_obs, n_t) matrix. These diagonals are then used by the loss to reconstruct the full Hessian with appropriate dimensions and structure. Currently, full Hessians inferred by data are not supported.

Parameters
  • y – target matrix (n_obs, n_t)

  • y_hat – current target estimation matrix (n_obs, n_t)

  • iteration – current number of iteration, generally not needed

  • leaves_idx – leaves’ indexes for each observation in y, (n_obs, 1). This is needed for example by mbtr.losses.QuadraticQuantileLoss.

Returns

grad, hessian_diags tuple, each of which is a (n_obs, n_t) matrix

get_initial_guess(y)

The initial guess are the alpha quantiles of the target matrix y.

Parameters

y – target matrix of the training set

Returns

np.ndarray with initial guess

quantile_loss(y, q_hat)

Quantile loss function, a.k.a. pinball loss.

\[ \begin{align}\begin{aligned}\epsilon (y,\hat{q})_{\alpha} &= \hat{q}_{\alpha} - y\\\mathcal{L} (y,\hat{q})_{\alpha} &= \epsilon (y,\hat{q})_{\alpha} \left( I_{\epsilon_{\alpha}\geq 0} -\alpha \right)\end{aligned}\end{align} \]
Parameters
  • y – observations of the target on the traning set

  • q_hat – current estimation matrix of the quantiles

Returns

tree loss

required_pars = ['alphas']
class mbtr.losses.TimeSmoother(lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: mbtr.losses.Loss

Time-smoothing loss function. Penalizes the time-derivative of the predicted signal.

\[\mathcal{L} = \frac{1}{2}\Vert y-w\Vert_2^{2} + \frac{1}{2} w^T \left(\lambda_s D^T D + \lambda I \right) w\]

where \(D\) is the second order difference matrix

\[ \begin{align}\begin{aligned}D=\left[\begin{array}{rrrrrr} 1 & -2 & 1 & & &\\& 1 & -2 & 1 & &\\& & \ddots & \ddots & \ddots &\\& & & 1 & -2 & 1\\& & & & 1 & -2 & 1 \end{array}\right]\end{aligned}\end{align} \]

and \(\lambda_s\) is the coefficient for the quadratic penalization of time-derivatives. Required parameters: lambda_smooth: coefficient for the quadratic penalization of time-derivatives

static build_filter_mat(n)

Build the second order difference matrix

Parameters

n – target dimension

Returns

D, second order difference matrix

compute_fast_H_inv(n)
required_pars = ['lambda_smooth']
set_dimension(n_dim)

Initialize all the properties which depends on the dimension of the target

Parameters

n_dims – dimension of the target

Returns

None

update_smoothing_mat(smoothing_weights)

mbtr.mbtr module

class mbtr.mbtr.MBT(n_boosts: int = 20, early_stopping_rounds: int = 3, learning_rate: float = 0.1, val_ratio: int = 0, n_q: int = 10, min_leaf: int = 100, loss_type: str = 'mse', lambda_weights: float = 0.1, lambda_leaves: float = 0.1, verbose: int = 0, **loss_kwargs)

Bases: object

Multivariate Boosted Tree class. Fits a multivariate tree using boosting.

Parameters
  • n_boosts – maximum number of boosting rounds. Default: 20

  • early_stopping_rounds – if the total loss is non-decreasing after early_stopping_rounds, stop training. The final model is the one which achieved the lowest loss up to the final iteration. Default: 3.

  • learning_rate – in [0, 1]. A learning rate < 1 helps reducing overfitting. Default: 0.1.

  • val_ratio – in [0,1]. If provided, the early stop is triggered by the loss computed on a validation set, randomly extracted from the training set. The length of the validation set is val_ratio * len (training set). Default: 0.

  • n_q – number of quantiles for the split search. Default: 10.

  • min_leaf – minimum number of observations in one leaf. This parameter greatly affect generalization abilities. Default: 100.

  • loss_type

    loss type for choosing the best splits. Currently the following losses are implemented:

    mse: mean squared error loss, a.k.a. L2, ordinary least squares

    time_smoother: mse with an additional penalization on the second order differences of the response function. Requires to pass also lambda_smooth parameter.

    latent_variable: generate response function of dimension n_t from an arbitrary linear combination of n_r responses. This requires to pass also S and precision pars.

    linear_regression: mse with linear response function. Using this loss function, when calling fit and predict methods, one must also pass x_lr as additional argument, which is the matrix of features used to train the linear response inside the leaf (which can be different from the features used to grow the tree, x).

    fourier: mse with linear response function, fitted on the first n_harmonics (where the fundamental has wave-lenght equal to the target output). This requires to pass also the n_harmonics parameter.

    quantile: quantile loss function, a.k.a. pinball loss. This requires to pass also the alphas parameter, a list of quantiles to be fitted.

    quadratic_quantile: quadratic quantile loss function tailored for trees. It has a non-discontinuos derivative. This requires to pass also the alphas parameter, a list of quantiles to be fitted.

  • lambda_weights – coefficient for the quadratic regularization of the response’s parameters. Default: 0.1

  • lambda_leaves – coefficient for the quadratic regularization of the total number of leaves. This is only used when the Tree is used as a weak learner by MBT. Default: 0.1

  • verbose – in {0,1}. If set to 1, the MBT return fitting information at each iteration.

  • loss_kwargs – possible additional arguments for the loss function

fit(x, y, do_plot=False, x_lr=None)

Fits an MBT using the features specified in the matrix \(x\in\mathbb{R}^{n_{obs} \times n_{f}}\), in order to predict the targets in the matrix \(y\in\mathbb{R}^{n_{obs} \times n_{t}}\), where \(n_{obs}\) is the number of observations, \(n_{f}\) the number of features and \(n_{t}\) the dimension of the target.

Parameters
  • x – feature matrix, np.ndarray.

  • y – target matrix, np.ndarray.

  • x_lr – features for fitting the linear response inside the leaves. This is only required if a LinearLoss is being used.

predict(x, n=None, x_lr=None)

Predicts the target based on the feature matrix x (and linear regression features x_lr).

Parameters
  • x – feature matrix, np.ndarray.

  • n – predict up to the nth fitted tree. If None, predict all the trees. Default: None

  • x_lr – linear regression feature matrix, np.ndarray. Only required if LinearLoss has been used.

Returns

target’s predictions

class mbtr.mbtr.Tree(n_q: int = 10, min_leaf: int = 100, loss_type: str = 'mse', lambda_weights: float = 0.1, lambda_leaves: float = 0.1, **loss_kwargs)

Bases: object

Tree class. Fits both univarite and multivariate targets. It implements histogram search for the decision of the splitting points.

Parameters
  • n_q – number of quantiles for the split search

  • min_leaf – minimum number of observations in one leaf. This parameter greatly affect generalization abilities.

  • loss_type

    loss type for choosing the best splits. Currently the following losses are implemented:

    mse: mean squared error loss, a.k.a. L2, ordinary least squares

    time_smoother: mse with an additional penalization on the second order differences of the response function. Requires to pass also lambda_smooth parameter.

    latent_variable: generate response function of dimension n_t from an arbitrary linear combination of n_r responses. This requires to pass also S and precision pars.

    linear_regression: mse with linear response function. Using this loss function, when calling fit and predict methods, one must also pass x_lr as additional argument, which is the matrix of features used to train the linear response inside the leaf (which can be different from the features used to grow the tree, x).

    fourier: mse with linear response function, fitted on the first n_harmonics (where the fundamental has wave-lenght equal to the target output). This requires to pass also the n_harmonics parameter.

    quantile: quantile loss function, a.k.a. pinball loss. This requires to pass also the alphas parameter, a list of quantiles to be fitted.

    quadratic_quantile: quadratic quantile loss function tailored for trees. It has a non-discontinuos derivative. This requires to pass also the alphas parameter, a list of quantiles to be fitted.

  • lambda_weights – coefficient for the quadratic regularization of the response’s parameters

  • lambda_leaves – coefficient for the quadratic regularization of the total number of leaves. This is only used

when the Tree is used as a weak learner by MBT. :param loss_kwargs: possible additional arguments for the loss function

compute_loss(G2_left, G2_right, H_left, H_right, j)
fit(x, y, hessian=None, learning_rate=1.0, x_lr=None)

Fits a tree using the features specified in the matrix \(x\in\mathbb{R}^{n_{obs} \times n_{f}}\), in order to predict the targets in the matrix \(y\in\mathbb{R}^{n_{obs} \times n_{t}}\), where \(n_{obs}\) is the number of observations, \(n_{f}\) the number of features and \(n_{t}\) the dimension of the target.

Parameters
  • x – feature matrix, np.ndarray.

  • y – target matrix, np.ndarray.

  • hessian – diagonals of the hessians \(\in\mathbb{R}^{n_{obs} \times n_{t}}\). If None, each entry is set equal to one (this will result in the default behaviour under MSE loss). Default: None

  • learning_rate – learning rate used by the MBT instance. Default: 1

  • x_lr – features for fitting the linear response inside the leaves. This is only required if a LinearLoss is being used.

predict(x, x_lr=None)

Predicts the target based on the feature matrix x (and linear regression features x_lr).

param: x: feature matrix, np.ndarray. param: x_lr: linear regression feature matrix, np.ndarray.

Returns

target’s predictions

mbtr.mbtr.bin_sums(y, edges, x)
mbtr.mbtr.leaf_stats(y, edges, x, order)

mbtr.utils module

class mbtr.utils.LightGBMMISO(n_estimators, lgb_pars=None)

Bases: object

fit(x, y)
predict(x)
mbtr.utils.check_pars(required_pars, **kwargs)
mbtr.utils.download_dataset()
mbtr.utils.load_dataset()
mbtr.utils.set_figure(size, subplots=1, 1, context='paper', style='darkgrid', font_scale=1, l=0.2, w=0.1, h=0.1, b=0.1)

Module contents