autots.evaluator package

Submodules

autots.evaluator.auto_model module

Mid-level helper functions for AutoTS.

autots.evaluator.auto_model.ModelMonster(model: str, parameters: dict = {}, frequency: str = 'infer', prediction_interval: float = 0.9, holiday_country: str = 'US', startTimeStamps=None, forecast_length: int = 14, random_seed: int = 2020, verbose: int = 0, n_jobs: int = None, **kwargs)

Directs strings and parameters to appropriate model objects.

Parameters
  • model (str) – Name of Model Function

  • parameters (dict) – Dictionary of parameters to pass through to model

autots.evaluator.auto_model.ModelPrediction(df_train, forecast_length: int, transformation_dict: dict, model_str: str, parameter_dict: dict, frequency: str = 'infer', prediction_interval: float = 0.9, no_negatives: bool = False, constraint: float = None, future_regressor_train=[], future_regressor_forecast=[], holiday_country: str = 'US', startTimeStamps=None, grouping_ids=None, random_seed: int = 2020, verbose: int = 0, n_jobs: int = None)

Feed parameters into modeling pipeline

Parameters
  • df_train (pandas.DataFrame) – numeric training dataset of DatetimeIndex and series as cols

  • forecast_length (int) – number of periods to forecast

  • transformation_dict (dict) – a dictionary of outlier, fillNA, and transformation methods to be used

  • model_str (str) – a string to be direct to the appropriate model, used in ModelMonster

  • frequency (str) – str representing frequency alias of time series

  • prediction_interval (float) – width of errors (note: rarely do the intervals accurately match the % asked for…)

  • no_negatives (bool) – whether to force all forecasts to be > 0

  • constraint (float) – when not None, use this value * data st dev above max or below min for constraining forecast values.

  • future_regressor_train (pd.Series) – with datetime index, of known in advance data, section matching train data

  • future_regressor_forecast (pd.Series) – with datetime index, of known in advance data, section matching test data

  • holiday_country (str) – passed through to holiday package, used by a few models as 0/1 regressor.

  • startTimeStamps (pd.Series) – index (series_ids), columns (Datetime of First start of series)

  • n_jobs (int) – number of processes

Returns

Prediction from AutoTS model object

Return type

PredictionObject (autots.PredictionObject)

autots.evaluator.auto_model.NewGeneticTemplate(model_results, submitted_parameters, sort_column: str = 'smape_weighted', sort_ascending: bool = True, max_results: int = 50, max_per_model_class: int = 5, top_n: int = 50, template_cols: list = ['Model', 'ModelParameters', 'TransformationParameters', 'Ensemble'], transformer_list: dict = {}, transformer_max_depth: int = 8)

Return new template given old template with model accuracies.

Parameters
  • model_results (pandas.DataFrame) – models that have actually been run

  • submitted_paramters (pandas.DataFrame) – models tried (may have returned different parameters to results)

autots.evaluator.auto_model.PredictWitch(template, df_train, forecast_length: int, frequency: str = 'infer', prediction_interval: float = 0.9, no_negatives: bool = False, constraint: float = None, future_regressor_train=[], future_regressor_forecast=[], holiday_country: str = 'US', startTimeStamps=None, grouping_ids=None, random_seed: int = 2020, verbose: int = 0, n_jobs: int = None, template_cols: list = ['Model', 'ModelParameters', 'TransformationParameters', 'Ensemble'], horizontal_subset: list = None)

Takes numeric data, returns numeric forecasts.

Only one model (albeit potentially an ensemble)!

Well, she turned me into a newt. A newt? I got better. -Python

Parameters
  • df_train (pandas.DataFrame) – numeric training dataset of DatetimeIndex and series as cols

  • forecast_length (int) – number of periods to forecast

  • transformation_dict (dict) – a dictionary of outlier, fillNA, and transformation methods to be used

  • model_str (str) – a string to be direct to the appropriate model, used in ModelMonster

  • frequency (str) – str representing frequency alias of time series

  • prediction_interval (float) – width of errors (note: rarely do the intervals accurately match the % asked for…)

  • no_negatives (bool) – whether to force all forecasts to be > 0

  • constraint (float) – when not None, use this value * data st dev above max or below min for constraining forecast values.

  • future_regressor_train (pd.Series) – with datetime index, of known in advance data, section matching train data

  • future_regressor_forecast (pd.Series) – with datetime index, of known in advance data, section matching test data

  • holiday_country (str) – passed through to holiday package, used by a few models as 0/1 regressor.

  • startTimeStamps (pd.Series) – index (series_ids), columns (Datetime of First start of series)

  • template_cols (list) – column names of columns used as model template

  • horizontal_subset (list) – columns of df_train to use for forecast, meant for horizontal ensembling

Returns

Prediction from AutoTS model object):

Return type

PredictionObject (autots.PredictionObject)

autots.evaluator.auto_model.RandomTemplate(n: int = 10, model_list: list = ['ZeroesNaive', 'LastValueNaive', 'AverageValueNaive', 'GLS', 'GLM', 'ETS', 'ARIMA', 'FBProphet', 'RollingRegression', 'GluonTS', 'UnobservedComponents', 'VARMAX', 'VECM', 'DynamicFactor'], transformer_list: dict = {}, transformer_max_depth: int = 8)

Returns a template dataframe of randomly generated transformations, models, and hyperparameters.

Parameters

n (int) – number of random models to return

class autots.evaluator.auto_model.TemplateEvalObject(model_results=Empty DataFrame Columns: [] Index: [], per_timestamp_smape=Empty DataFrame Columns: [] Index: [], per_series_mae=Empty DataFrame Columns: [] Index: [], per_series_spl=Empty DataFrame Columns: [] Index: [], per_series_rmse1=Empty DataFrame Columns: [] Index: [], per_series_rmse2=Empty DataFrame Columns: [] Index: [], model_count: int = 0)

Bases: object

Object to contain all your failures!.

concat(another_eval)

Merge another TemplateEvalObject onto this one.

save(filename)

Save results to a file.

autots.evaluator.auto_model.TemplateWizard(template, df_train, df_test, weights, model_count: int = 0, ensemble: str = True, forecast_length: int = 14, frequency: str = 'infer', prediction_interval: float = 0.9, no_negatives: bool = False, constraint: float = None, future_regressor_train=[], future_regressor_forecast=[], holiday_country: str = 'US', startTimeStamps=None, random_seed: int = 2020, verbose: int = 0, n_jobs: int = None, validation_round: int = 0, current_generation: int = 0, max_generations: int = 0, model_interrupt: bool = False, grouping_ids=None, template_cols: list = ['Model', 'ModelParameters', 'TransformationParameters', 'Ensemble'], traceback: bool = False)

Take Template, returns Results.

There are some who call me… Tim. - Python

Parameters
  • template (pandas.DataFrame) – containing model str, and json of transformations and hyperparamters

  • df_train (pandas.DataFrame) – numeric training dataset of DatetimeIndex and series as cols

  • df_test (pandas.DataFrame) – dataframe of actual values of (forecast length * n series)

  • weights (dict) – key = column/series_id, value = weight

  • ensemble (str) – desc of ensemble types to prepare metric collection

  • forecast_length (int) – number of periods to forecast

  • transformation_dict (dict) – a dictionary of outlier, fillNA, and transformation methods to be used

  • model_str (str) – a string to be direct to the appropriate model, used in ModelMonster

  • frequency (str) – str representing frequency alias of time series

  • prediction_interval (float) – width of errors (note: rarely do the intervals accurately match the % asked for…)

  • no_negatives (bool) – whether to force all forecasts to be > 0

  • constraint (float) – when not None, use this value * data st dev above max or below min for constraining forecast values.

  • future_regressor_train (pd.Series) – with datetime index, of known in advance data, section matching train data

  • future_regressor_forecast (pd.Series) – with datetime index, of known in advance data, section matching test data

  • holiday_country (str) – passed through to holiday package, used by a few models as 0/1 regressor.

  • startTimeStamps (pd.Series) – index (series_ids), columns (Datetime of First start of series)

  • validation_round (int) – int passed to record current validation.

  • current_generation (int) – info to pass to print statements

  • max_generations (int) – info to pass to print statements

  • model_interrupt (bool) – if True, keyboard interrupts are caught and only break current model eval.

  • template_cols (list) – column names of columns used as model template

  • traceback (bool) – include tracebook over just error representation

Returns

TemplateEvalObject

autots.evaluator.auto_model.UniqueTemplates(existing_templates, new_possibilities, selection_cols: list = ['Model', 'ModelParameters', 'TransformationParameters', 'Ensemble'])

Returns unique dataframe rows from new_possiblities not in existing_templates.

Parameters

selection_cols (list) – list of column namess to use to judge uniqueness/match on

autots.evaluator.auto_model.create_model_id(model_str: str, parameter_dict: dict = {}, transformation_dict: dict = {})

Create a hash ID which should be unique to the model parameters.

autots.evaluator.auto_model.dict_recombination(a: dict, b: dict)

Recombine two dictionaries with identical keys. Return new dict.

autots.evaluator.auto_model.generate_score(model_results, metric_weighting: dict = {}, prediction_interval: float = 0.9)

Generate score based on relative accuracies.

autots.evaluator.auto_model.trans_dict_recomb(dict_array)

Recombine two transformation param dictionaries from array of dicts.

autots.evaluator.auto_model.unpack_ensemble_models(template, template_cols: list = ['Model', 'ModelParameters', 'TransformationParameters', 'Ensemble'], keep_ensemble: bool = True, recursive: bool = False)

Take ensemble models from template and add as new rows.

Parameters
  • template (pd.DataFrame) – AutoTS template containing template_cols

  • keep_ensemble (bool) – if False, drop row containing original ensemble

  • recursive (bool) – if True, unnest ensembles of ensembles…

autots.evaluator.auto_model.validation_aggregation(validation_results)

Aggregate a TemplateEvalObject.

autots.evaluator.auto_ts module

Higher-level backbone of auto time series modeling.

class autots.evaluator.auto_ts.AutoTS(forecast_length: int = 14, frequency: str = 'infer', prediction_interval: float = 0.9, max_generations: int = 20, no_negatives: bool = False, constraint: float = None, ensemble: str = 'auto', initial_template: str = 'General+Random', random_seed: int = 2020, holiday_country: str = 'US', subset: int = None, aggfunc: str = 'first', na_tolerance: float = 1, metric_weighting: dict = {'containment_weighting': 0, 'contour_weighting': 1, 'mae_weighting': 2, 'rmse_weighting': 2, 'runtime_weighting': 0, 'smape_weighting': 10, 'spl_weighting': 2}, drop_most_recent: int = 0, drop_data_older_than_periods: int = 100000, model_list: str = 'default', transformer_list: dict = 'fast', transformer_max_depth: int = 6, num_validations: int = 2, models_to_validate: float = 0.15, max_per_model_class: int = None, validation_method: str = 'backwards', min_allowed_train_percent: float = 0.5, remove_leading_zeroes: bool = False, prefill_na: str = None, model_interrupt: bool = False, verbose: int = 1, n_jobs: int = None)

Bases: object

Automate time series modeling using a genetic algorithm.

Parameters
  • forecast_length (int) – number of periods over which to evaluate forecast. Can be overriden later in .predict().

  • frequency (str) – ‘infer’ or a specific pandas datetime offset. Can be used to force rollup of data (ie daily input, but frequency ‘M’ will rollup to monthly).

  • prediction_interval (float) – 0-1, uncertainty range for upper and lower forecasts. Adjust range, but rarely matches actual containment.

  • max_generations (int) – number of genetic algorithms generations to run. More runs = longer runtime, generally better accuracy.

  • no_negatives (bool) – if True, all negative predictions are rounded up to 0.

  • constraint (float) – when not None, use this value * data st dev above max or below min for constraining forecast values. Applied to point forecast only, not upper/lower forecasts.

  • ensemble (str) – None or list or comma-separated string containing: ‘auto’, ‘simple’, ‘distance’, ‘horizontal-max’, ‘probabilistic-max’, “hdist”

  • initial_template (str) – ‘Random’ - randomly generates starting template, ‘General’ uses template included in package, ‘General+Random’ - both of previous. Also can be overriden with self.import_template()

  • random_seed (int) – random seed allows (slightly) more consistent results.

  • holiday_country (str) – passed through to Holidays package for some models.

  • subset (int) – maximum number of series to evaluate at once. Useful to speed evaluation when many series are input.

  • aggfunc (str) – if data is to be rolled up to a higher frequency (daily -> monthly) or duplicate timestamps are included. Default ‘first’ removes duplicates, for rollup try ‘mean’ or np.sum. Beware numeric aggregations like ‘mean’ will not work with non-numeric inputs.

  • na_tolerance (float) – 0 to 1. Series are dropped if they have more than this percent NaN. 0.95 here would allow series containing up to 95% NaN values.

  • metric_weighting (dict) – weights to assign to metrics, effecting how the ranking score is generated.

  • drop_most_recent (int) – option to drop n most recent data points. Useful, say, for monthly sales data where the current (unfinished) month is included. occurs after any aggregration is applied, so will be whatever is specified by frequency, will drop n frequencies

  • drop_data_older_than_periods (int) – take only the n most recent timestamps

  • model_list (list) – str alias or list of names of model objects to use

  • transformer_list (list) – list of transformers to use, or dict of transformer:probability. Note this does not apply to initial templates.

  • transformer_max_depth (int) – maximum number of sequential transformers to generate for new Random Transformers. Fewer will be faster.

  • num_validations (int) – number of cross validations to perform. 0 for just train/test on final split.

  • models_to_validate (int) – top n models to pass through to cross validation. Or float in 0 to 1 as % of tried. 0.99 is forced to 100% validation. 1 evaluates just 1 model. If horizontal or probabilistic ensemble, then additional min per_series models above the number here may be added to validation.

  • max_per_model_class (int) – of the models_to_validate what is the maximum to pass from any one model class/family.

  • validation_method (str) – ‘even’, ‘backwards’, or ‘seasonal n’ where n is an integer of seasonal ‘backwards’ is better for recency and for shorter training sets ‘even’ splits the data into equally-sized slices best for more consistent data ‘seasonal n’ for example ‘seasonal 364’ would test all data on each previous year of the forecast_length that would immediately follow the training data.

  • min_allowed_train_percent (float) – percent of forecast length to allow as min training, else raises error. 0.5 with a forecast length of 10 would mean 5 training points are mandated, for a total of 15 points. Useful in (unrecommended) cases where forecast_length > training length.

  • remove_leading_zeroes (bool) – replace leading zeroes with NaN. Useful in data where initial zeroes mean data collection hasn’t started yet.

  • prefill_na (str) – value to input to fill all NaNs with. Leaving as None and allowing model interpolation is recommended. None, 0, ‘mean’, or ‘median’. 0 may be useful in for examples sales cases where all NaN can be assumed equal to zero.

  • model_interrupt (bool) – if False, KeyboardInterrupts quit entire program. if True, KeyboardInterrupts attempt to only quit current model. if True, recommend use in conjunction with verbose > 0 and result_file in the event of accidental complete termination.

  • verbose (int) – setting to 0 or lower should reduce most output. Higher numbers give more output.

  • n_jobs (int) – Number of cores available to pass to parallel processing. A joblib context manager can be used instead (pass None in this case). Also ‘auto’.

best_model

DataFrame containing template for the best ranked model

Type

pandas.DataFrame

regression_check

If True, the best_model uses an input ‘User’ future_regressor

Type

bool

export_template(filename=None, models: str = 'best', n: int = 5, max_per_model_class: int = None, include_results: bool = False)

Export top results as a reusable template.

Parameters
  • filename (str) – ‘csv’ or ‘json’ (in filename). None to return a dataframe and not write a file.

  • models (str) – ‘best’ or ‘all’

  • n (int) – if models = ‘best’, how many n-best to export

  • max_per_model_class (int) – if models = ‘best’, the max number of each model class to include in template

  • include_results (bool) – whether to include performance metrics

failure_rate(result_set: str = 'initial')

Return fraction of models passing with exceptions.

Parameters

result_set (str, optional) – ‘validation’ or ‘initial’. Defaults to ‘initial’.

Returns

float.

fit(df, date_col: str = None, value_col: str = None, id_col: str = None, future_regressor=[], weights: dict = {}, result_file: str = None, grouping_ids=None)

Train algorithm given data supplied.

Parameters
  • df (pandas.DataFrame) – Datetime Indexed dataframe of series, or dataframe of three columns as below.

  • date_col (str) – name of datetime column

  • value_col (str) – name of column containing the data of series.

  • id_col (str) – name of column identifying different series.

  • future_regressor (numpy.Array) – single external regressor matching train.index

  • weights (dict) – {‘colname1’: 2, ‘colname2’: 5} - increase importance of a series in metric evaluation. Any left blank assumed to have weight of 1. pass the alias ‘mean’ as a str ie weights=’mean’ to automatically use the mean value of a series as its weight available aliases: mean, median, min, max

  • result_file (str) – results saved on each new generation. Does not include validation rounds. “.csv” save model results table. “.pickle” saves full object, including ensemble information.

  • grouping_ids (dict) – currently a one-level dict containing series_id:group_id mapping. used in 0.2.x but not 0.3.x+ versions. retained for potential future use

import_results(filename)

Add results from another run on the same data.

Input can be filename with .csv or .pickle. or can be a DataFrame of model results or a full TemplateEvalObject

import_template(filename: str, method: str = 'add_on', enforce_model_list: bool = True)

Import a previously exported template of model parameters. Must be done before the AutoTS object is .fit().

Parameters
  • filename (str) – file location (or a pd.DataFrame already loaded)

  • method (str) – ‘add_on’ or ‘only’ - “add_on” keeps initial_template generated in init. “only” uses only this template.

  • enforce_model_list (bool) – if True, remove model types not in model_list

predict(forecast_length: int = 'self', prediction_interval: float = 'self', future_regressor=[], hierarchy=None, just_point_forecast: bool = False, verbose: int = 'self')

Generate forecast data immediately following dates of index supplied to .fit().

Parameters
  • forecast_length (int) – Number of periods of data to forecast ahead

  • prediction_interval (float) – interval of upper/lower forecasts. defaults to ‘self’ ie the interval specified in __init__() if prediction_interval is a list, then returns a dict of forecast objects.

  • future_regressor (numpy.Array) – additional regressor

  • hierarchy – Not yet implemented

  • just_point_forecast (bool) – If True, return a pandas.DataFrame of just point forecasts

Returns

Either a PredictionObject of forecasts and metadata, or if just_point_forecast == True, a dataframe of point forecasts

results(result_set: str = 'initial')

Convenience function to return tested models table.

Parameters

result_set (str) – ‘validation’ or ‘initial’

class autots.evaluator.auto_ts.AutoTSIntervals

Bases: object

Autots looped to test multiple prediction intervals. Experimental.

Runs max_generations on first prediction interval, then validates on remainder. Most args are passed through to AutoTS().

Parameters
  • interval_models_to_validate (int) – number of models to validate on each prediction interval.

  • import_results (str) – results from run on same data to load, filename.pickle. Currently result_file and import only save/load initial run, no validations.

fit(prediction_intervals, forecast_length, df_long, max_generations, num_validations, validation_method, models_to_validate, interval_models_to_validate, date_col, value_col, id_col=None, import_template=None, import_method='only', import_results=None, result_file=None, model_list='all', metric_weighting: dict = {'containment_weighting': 0, 'contour_weighting': 0, 'mae_weighting': 0, 'rmse_weighting': 1, 'runtime_weighting': 0, 'smape_weighting': 1, 'spl_weighting': 10}, weights: dict = {}, grouping_ids=None, future_regressor=[], model_interrupt: bool = False, constraint=2, no_negatives=False, remove_leading_zeroes=False, random_seed=2020)

Train and find best.

predict(future_regressor=[], verbose: int = 'self') → dict

Generate forecasts after training complete.

autots.evaluator.auto_ts.error_correlations(all_result, result: str = 'corr')

Onehot encode AutoTS result df and return df or correlation with errors.

Parameters
  • all_results (pandas.DataFrame) – AutoTS model_results df

  • result (str) – whether to return ‘df’, ‘corr’, ‘poly corr’ with errors

autots.evaluator.auto_ts.fake_regressor(df, forecast_length: int = 14, date_col: str = None, value_col: str = None, id_col: str = None, frequency: str = 'infer', aggfunc: str = 'first', drop_most_recent: int = 0, na_tolerance: float = 0.95, drop_data_older_than_periods: int = 100000, dimensions: int = 1, verbose: int = 0)

Create a fake regressor of random numbers for testing purposes.

autots.evaluator.metrics module

Tools for calculating forecast errors.

class autots.evaluator.metrics.EvalObject(model_name: str = 'Uninitiated', per_series_metrics=nan, per_timestamp=nan, avg_metrics=nan, avg_metrics_weighted=nan)

Bases: object

Object to contain all your failures!.

autots.evaluator.metrics.PredictionEval(PredictionObject, actual, series_weights: dict = {}, df_train=nan, per_timestamp_errors: bool = False, dist_n: int = None)

Evalute prediction against test actual.

Parameters
  • PredictionObject (autots.PredictionObject) – Prediction from AutoTS model object

  • actual (pd.DataFrame) – dataframe of actual values of (forecast length * n series)

  • series_weights (dict) – key = column/series_id, value = weight

  • per_timestamp (bool) – whether to calculate and return per timestamp direction errors

  • dist_n (int) – if not None, calculates two part rmse on head(n) and tail(remainder) of forecast.

autots.evaluator.metrics.SPL(A, F, df_train, quantile)

Scaled pinball loss.

autots.evaluator.metrics.containment(lower_forecast, upper_forecast, actual)

Expects two, 2-D numpy arrays of forecast_length * n series.

Returns a 1-D array of results in len n series

Parameters
  • actual (numpy.array) – known true values

  • forecast (numpy.array) – predicted values

autots.evaluator.metrics.contour(A, F)

A measure of how well the actual and forecast follow the same pattern of change. Note: If actual values are unchanging, will match positive changing forecasts. Expects two, 2-D numpy arrays of forecast_length * n series Returns a 1-D array of results in len n series

Parameters
  • A (numpy.array) – known true values

  • F (numpy.array) – predicted values

autots.evaluator.metrics.mae(A, F)

Expects two, 2-D numpy arrays of forecast_length * n series.

Returns a 1-D array of results in len n series

Parameters
  • A (numpy.array) – known true values

  • F (numpy.array) – predicted values

autots.evaluator.metrics.pinball_loss(A, F, quantile)

Bigger is bad-er.

autots.evaluator.metrics.rmse(actual, forecast)

Expects two, 2-D numpy arrays of forecast_length * n series.

Returns a 1-D array of results in len n series

Parameters
  • actual (numpy.array) – known true values

  • forecast (numpy.array) – predicted values

autots.evaluator.metrics.smape(actual, forecast)

Expect two, 2-D numpy arrays of forecast_length * n series. Allows NaN in actuals, and corresponding NaN in forecast, but not unmatched NaN in forecast Also doesn’t like zeroes in either forecast or actual - results in poor error value even if forecast is accurate

Returns a 1-D array of results in len n series

Parameters
  • actual (numpy.array) – known true values

  • forecast (numpy.array) – predicted values

References

https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error

Module contents

Model Evaluators