pydgn.evaluation
evaluation.config
- class pydgn.evaluation.config.Config(config_dict: dict)
Bases:
object
Simple class to manage the configuration dictionary as a Python object with fields.
- Parameters
config_dict (dict) – the configuration dictionary
- items() collections.abc.ItemsView
Invokes the items() method of the configuration dictionary
- Returns
a list of (key, value) pairs
- keys() collections.abc.KeysView
Invokes the keys() method of the configuration dictionary
- Returns
the set of keys in the dictionary
evaluation.evaluator
- class pydgn.evaluation.evaluator.RiskAssesser(outer_folds: int, inner_folds: int, experiment_class: Callable[[...], pydgn.experiment.experiment.Experiment], exp_path: str, splits_filepath: str, model_configs: Union[pydgn.evaluation.grid.Grid, pydgn.evaluation.random_search.RandomSearch], final_training_runs: int, higher_is_better: bool, gpus_per_task: float, base_seed: int = 42)
Bases:
object
Class implementing a K-Fold technique to do Risk Assessment (estimate of the true generalization performances) and K-Fold Model Selection (select the best hyper-parameters for each external fold
- Parameters
outer_folds (int) – The number K of outer TEST folds. You should have generated the splits accordingly
outer_folds – The number K of inner VALIDATION folds. You should have generated the splits accordingly
experiment_class (Callable[…,
Experiment
]) – the experiment class to be instantiatedexp_path (str) – The folder in which to store all results
splits_filepath (str) – The splits filepath with additional meta information
model_configs (Union[
Grid
,RandomSearch
]) – an object storing all possible model configurations, e.g. config.base.Gridfinal_training_runs (int) – no of final training runs to mitigate bad initializations
higher_is_better (bool) – whether or not the best model for each external fold should be selected by higher or lower score values
gpus_per_task (float) – Number of gpus to assign to each experiment. Can be <
1
.base_seed (int) – Seed used to generate experiments seeds. Used to replicate results. Default is
42
- model_selection(kfold_folder: str, outer_k: int, debug: bool)
Performs model selection.
- Parameters
kfold_folder – The root folder for model selection
outer_k – the current outer fold to consider
debug – if
True
, sequential execution is performed and logs are printed to screen
- process_config(config_folder: str, config: pydgn.evaluation.config.Config)
Computes the best configuration for each external fold and stores it into a file.
- Parameters
config_folder (str) –
config (
Config
) – the configuration object
- process_final_runs(outer_k: int)
Computes the average scores for the final runs of a specific outer fold
- Parameters
outer_k (int) – id of the outer fold from 0 to K-1
- process_inner_results(folder: str, no_configurations: int)
Chooses the best hyper-parameters configuration using the HIGHEST validation mean score.
- Parameters
folder (str) – a folder which holds all configurations results after K INNER folds
no_configurations (int) – number of possible configurations
- process_outer_results()
Aggregates Outer Folds results and compute Training and Test mean/std
- risk_assessment(debug: bool)
Performs risk assessment to evaluate the performances of a model.
- Parameters
debug – if
True
, sequential execution is performed and logs are printed to screen
- run_final_model(outer_k: int, debug: bool)
Performs the final runs once the best model for outer fold
outer_k
has been chosen.- Parameters
outer_k (int) – the current outer fold to consider
debug (bool) – if
True
, sequential execution is performed and logs are printed to screen
- wait_configs()
Waits for configurations to terminate and updates the state of the progress manager
evaluation.grid
- class pydgn.evaluation.grid.Grid(configs_dict: dict)
Bases:
object
Class that implements grid-search. It computes all possible configurations starting from a suitable config file.
- Parameters
configs_dict (dict) – the configuration dictionary specifying the different configurations to try
- property exp_name: str
Computes the name of the root folder
- Returns
the name of the root folder as made of
EXP-NAME_DATASET-NAME
- property num_configs: int
Computes the number of configurations to try during model selection
- Returns
the number of configurations
evaluation.random_search
- class pydgn.evaluation.random_search.RandomSearch(configs_dict: dict)
Bases:
pydgn.evaluation.grid.Grid
Class that implements random-search. It computes all possible configurations starting from a suitable config file.
- Parameters
configs_dict (dict) – the configuration dictionary specifying the different configurations to try
evaluation.util
- class pydgn.evaluation.util.ProgressManager(outer_folds, inner_folds, no_configs, final_runs, show=True)
Bases:
object
Class that is responsible for drawing progress bars.
- Parameters
outer_folds (int) – number of external folds for model assessment
inner_folds (int) – number of internal folds for model selection
no_configs (int) – number of possible configurations in model selection
final_runs (int) – number of final runs per outer fold once the best model has been selected
show (bool) – whether to show the progress bar or not. Default is
True
- refresh()
Refreshes the progress bar
- show_header()
Prints the header of the progress bar
- pydgn.evaluation.util.choice(*args)
- pydgn.evaluation.util.clear_screen()
Clears the CLI interface.
- pydgn.evaluation.util.loguniform(*args)
Performs a log-uniform random selection.
- Parameters
*args – a tuple of (log min, log max, [base]) to use. Base 10 is used if the third argument is not available.
- Returns
a randomly chosen value
- pydgn.evaluation.util.normal(*args)
- pydgn.evaluation.util.randint(*args)
- pydgn.evaluation.util.return_class_and_args(config: dict, key: str, return_class_name: bool = False) Tuple[Callable[[...], object], dict]
Returns the class and arguments associated to a specific key in the configuration file.
- Parameters
config (dict) – the configuration dictionary
key (str) – a string representing a particular class in the configuration dictionary
return_class_name (bool) – if
True
, returns the class name as a string rather than the class object
- Returns
a tuple (class, dict of arguments), or (None, None) if the key is not present in the config dictionary
- pydgn.evaluation.util.set_gpus(num_gpus)
Sets the visible GPUS for the experiments according to the availability in terms of memory. Prioritize GPUs with less memory usage. Sets the
CUDA_DEVICE_ORDER
env variable toPCI_BUS_ID
andCUDA_VISIBLE_DEVICES
to the ordered list of GPU indices.- Parameters
num_gpus – maximum number of GPUs to use when launching experiments in parallel
- pydgn.evaluation.util.uniform(*args)