super_gradients.training.datasets package
Subpackages
- super_gradients.training.datasets.classification_datasets package
- super_gradients.training.datasets.dataset_interfaces package
- super_gradients.training.datasets.detection_datasets package
- super_gradients.training.datasets.segmentation_datasets package
- Submodules
- super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation module
- super_gradients.training.datasets.segmentation_datasets.coco_segmentation module
- super_gradients.training.datasets.segmentation_datasets.pascal_aug_segmentation module
- super_gradients.training.datasets.segmentation_datasets.pascal_voc_segmentation module
- super_gradients.training.datasets.segmentation_datasets.segmentation_dataset module
- Module contents
Submodules
super_gradients.training.datasets.all_datasets module
- exception super_gradients.training.datasets.all_datasets.DataSetDoesNotExistException[source]
Bases:
Exception
The requested dataset does not exist, or is not implemented.
- class super_gradients.training.datasets.all_datasets.SgLibraryDatasets[source]
Bases:
object
Holds all of the different library dataset dictionaries, by DL Task mapping
- Attributes:
CLASSIFICATION Dictionary of Classification Data sets OBJECT_DETECTION Dictionary of Object Detection Data sets SEMANTIC_SEGMENTATION Dictionary of Semantic Segmentation Data sets
- CLASSIFICATION = {'cifar_10': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.Cifar10DatasetInterface'>, 'cifar_100': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.Cifar100DatasetInterface'>, 'classification_dataset': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.ClassificationDatasetInterface'>, 'imagenet': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.ImageNetDatasetInterface'>, 'library_dataset': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.LibraryDatasetInterface'>, 'test_dataset': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface'>, 'tiny_imagenet': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.TinyImageNetDatasetInterface'>}
- OBJECT_DETECTION = {'coco': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDetectionDatasetInterface'>, 'coco2014': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCo2014DetectionDatasetInterface'>}
- SEMANTIC_SEGMENTATION = {'coco': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoSegmentationDatasetInterface'>, 'pascal_aug': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.PascalAUG2012SegmentationDataSetInterface'>, 'pascal_voc': <class 'super_gradients.training.datasets.dataset_interfaces.dataset_interface.PascalVOC2012SegmentationDataSetInterface'>}
- static get_all_available_datasets() → Dict[str, List[str]][source]
Gets all the available datasets.
- static get_dataset(dl_task: str, dataset_name: str) → Type[super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface][source]
Get’s a dataset with a given name for a given deep learning task. examp: >>> SgLibraryDatasets.get_dataset(dl_task=’classification’, dataset_name=’cifar_100’) >>> <Cifar100DatasetInterface instance>
super_gradients.training.datasets.auto_augment module
RandAugment RandAugment is a variant of AutoAugment which randomly selects transformations
from AutoAugment to be applied on an image.
- RandomAugmentation Implementation adapted from:
https://github.com/rwightman/pytorch-image-models/blob/master/timm/data/auto_augment.py
- Papers:
RandAugment: Practical automated data augmentation… - https://arxiv.org/abs/1909.13719
- class super_gradients.training.datasets.auto_augment.AugmentOp(name, prob=0.5, magnitude=10, hparams=None)[source]
Bases:
object
single auto augment operations
- class super_gradients.training.datasets.auto_augment.RandAugment(ops, num_layers=2, choice_weights=None)[source]
Bases:
object
Random auto augment class, will select auto augment transforms according to probability weights for each op
- super_gradients.training.datasets.auto_augment.rand_augment_ops(magnitude=10, hparams=None, transforms=None)[source]
- super_gradients.training.datasets.auto_augment.rand_augment_transform(config_str, hparams)[source]
Create a RandAugment transform
- Parameters
config_str – String defining configuration of random augmentation. Consists of multiple sections separated by
dashes (‘-‘). The first section defines the specific variant of rand augment (currently only ‘rand’). The remaining sections, not order sepecific determine
‘m’ - integer magnitude of rand augment ‘n’ - integer num layers (number of transform ops selected per image) ‘w’ - integer probabiliy weight index (index of a set of weights to influence choice of op) ‘mstd’ - float std deviation of magnitude noise applied ‘inc’ - integer (bool), use augmentations that increase in severity with magnitude (default: 0)
Ex ‘rand-m9-n3-mstd0.5’ results in RandAugment with magnitude 9, num_layers 3, magnitude_std 0.5 ‘rand-mstd1-w0’ results in magnitude_std 1.0, weights 0, default magnitude of 10 and num_layers 2
- Parameters
hparams – Other hparams (kwargs) for the RandAugmentation scheme
- Returns
A PyTorch compatible Transform
super_gradients.training.datasets.data_augmentation module
- class super_gradients.training.datasets.data_augmentation.Lighting(alphastd, eigval=tensor([0.2175, 0.0188, 0.0045]), eigvec=tensor([[- 0.5675, 0.7192, 0.4009], [- 0.5808, - 0.0045, - 0.814], [- 0.5836, - 0.6948, 0.4203]]))[source]
Bases:
object
Lighting noise(AlexNet - style PCA - based noise) Taken from fastai Imagenet training - https://github.com/fastai/imagenet-fast/blob/faa0f9dfc9e8e058ffd07a248724bf384f526fae/imagenet_nv/fastai_imagenet.py#L103 To use:
training_params = {“imagenet_pca_aug”: 0.1}
Default training_params arg is 0.0 (“don’t use”)
0.1 is that default in the original paper
super_gradients.training.datasets.datasets_conf module
super_gradients.training.datasets.datasets_utils module
- class super_gradients.training.datasets.datasets_utils.AbstractCollateFunction[source]
Bases:
abc.ABC
A collate function (for torch DataLoader)
- class super_gradients.training.datasets.datasets_utils.AtomicInteger(value: int = 0)[source]
Bases:
object
- class super_gradients.training.datasets.datasets_utils.ComposedCollateFunction(functions: list)[source]
Bases:
super_gradients.training.datasets.datasets_utils.AbstractCollateFunction
A function (for torch DataLoader) which executes a sequence of sub collate functions
- class super_gradients.training.datasets.datasets_utils.DatasetStatisticsTensorboardLogger(writer: torch.utils.tensorboard.writer.SummaryWriter, summary_params: dict = {'max_batches': 30, 'plot_anchors_coverage': True, 'plot_box_size_distribution': True, 'plot_class_distribution': True, 'sample_images': 32})[source]
Bases:
object
- DEFAULT_SUMMARY_PARAMS = {'max_batches': 30, 'plot_anchors_coverage': True, 'plot_box_size_distribution': True, 'plot_class_distribution': True, 'sample_images': 32}
- analyze(data_loader: torch.utils.data.dataloader.DataLoader, dataset_params: dict, title: str, anchors: Optional[list] = None)[source]
- Parameters
data_loader – the dataset data loader
dataset_params – the dataset parameters
title – the title for this dataset (i.e. Coco 2017 test set)
anchors – the list of anchors used by the model. applicable only for detection datasets
- logger = <Logger super_gradients.training.datasets.datasets_utils (INFO)>
- class super_gradients.training.datasets.datasets_utils.MultiScaleCollateFunction(target_size: Optional[int] = None, min_image_size: Optional[int] = None, max_image_size: Optional[int] = None, image_size_steps: int = 32, change_frequency: int = 10)[source]
Bases:
super_gradients.training.datasets.datasets_utils.AbstractCollateFunction
a collate function to implement multi-scale data augmentation according to https://arxiv.org/pdf/1612.08242.pdf
- class super_gradients.training.datasets.datasets_utils.RandomResizedCropAndInterpolation(size, scale=(0.08, 1.0), ratio=(0.75, 1.3333333333333333), interpolation='default')[source]
Bases:
torchvision.transforms.transforms.RandomResizedCrop
Crop the given PIL Image to random size and aspect ratio with explicitly chosen or random interpolation.
A crop of random size (default: of 0.08 to 1.0) of the original size and a random aspect ratio (default: of 3/4 to 4/3) of the original aspect ratio is made. This crop is finally resized to given size. This is popularly used to train the Inception networks.
- Args:
size: expected output size of each edge scale: range of size of the origin size cropped ratio: range of aspect ratio of the origin aspect ratio cropped interpolation: Default: PIL.Image.BILINEAR
- forward(img)[source]
- Args:
img (PIL Image): Image to be cropped and resized.
- Returns:
PIL Image: Randomly cropped and resized image.
- training: bool
- super_gradients.training.datasets.datasets_utils.get_color_augmentation(rand_augment_config_string: str, color_jitter: tuple, crop_size=224, img_mean=[0.485, 0.456, 0.406])[source]
Returns color augmentation class. As these augmentation cannot work on top one another, only one is returned according to rand_augment_config_string :param rand_augment_config_string: string which defines the auto augment configurations. If none, color jitter will be returned. For possibile values see auto_augment.py :param color_jitter: tuple for color jitter value. :param crop_size: relevant only for auto augment :param img_mean: relevant only for auto augment :return: RandAugment transform or ColorJitter
- super_gradients.training.datasets.datasets_utils.get_mean_and_std(dataset)[source]
Compute the mean and std value of dataset.
- super_gradients.training.datasets.datasets_utils.get_mean_and_std_torch(data_dir=None, dataloader=None, num_workers=4, RandomResizeSize=224)[source]
A function for getting the mean and std of large datasets using pytorch dataloader and gpu functionality.
- Parameters
data_dir – String, path to none-library dataset folder. For example “/data/Imagenette” or “/data/TinyImagenet”
dataloader – a torch DataLoader, as it would feed the data into the trainer (including transforms etc).
RandomResizeSize – Int, the size of the RandomResizeCrop as it appears in the DataInterface (for example, for Imagenet,
this value should be 224). :return: 2 lists,mean and std, each one of len 3 (1 for each channel)
super_gradients.training.datasets.mixup module
Mixup and Cutmix
Papers: mixup: Beyond Empirical Risk Minimization (https://arxiv.org/abs/1710.09412)
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features (https://arxiv.org/abs/1905.04899)
Code Reference: CutMix: https://github.com/clovaai/CutMix-PyTorch CutMix by timm: https://github.com/rwightman/pytorch-image-models/timm
- class super_gradients.training.datasets.mixup.CollateMixup(mixup_alpha: float = 1.0, cutmix_alpha: float = 0.0, cutmix_minmax: Optional[List[float]] = None, prob: float = 1.0, switch_prob: float = 0.5, mode: str = 'batch', correct_lam: bool = True, label_smoothing: float = 0.1, num_classes: int = 1000)[source]
Bases:
object
Collate with Mixup/Cutmix that applies different params to each element or whole batch A Mixup impl that’s performed while collating the batches.
- super_gradients.training.datasets.mixup.cutmix_bbox_and_lam(img_shape: tuple, lam: float, ratio_minmax: Optional[Union[tuple, list]] = None, correct_lam: bool = True, count: Optional[int] = None)[source]
Generate bbox and apply lambda correction.
- super_gradients.training.datasets.mixup.mixup_target(target: torch.Tensor, num_classes: int, lam: float = 1.0, smoothing: float = 0.0, device: str = 'cuda')[source]
generate a smooth target (label) two-hot tensor to support the mixed images with different labels :param target: the targets tensor :param num_classes: number of classes (to set the final tensor size) :param lam: percentage of label a range [0, 1] in the mixing :param smoothing: the smoothing multiplier :param device: usable device [‘cuda’, ‘cpu’] :return:
- super_gradients.training.datasets.mixup.one_hot(x, num_classes, on_value=1.0, off_value=0.0, device='cuda')[source]
- super_gradients.training.datasets.mixup.rand_bbox(img_shape: tuple, lam: float, margin: float = 0.0, count: Optional[int] = None)[source]
Standard CutMix bounding-box Generates a random square bbox based on lambda value. This impl includes support for enforcing a border margin as percent of bbox dimensions.
- Parameters
img_shape – Image shape as tuple
lam – Cutmix lambda value
margin – Percentage of bbox dimension to enforce as margin (reduce amount of box outside image)
count – Number of bbox to generate
- super_gradients.training.datasets.mixup.rand_bbox_minmax(img_shape: tuple, minmax: Union[tuple, list], count: Optional[int] = None)[source]
Min-Max CutMix bounding-box Inspired by Darknet cutmix impl, generates a random rectangular bbox based on min/max percent values applied to each dimension of the input image.
Typical defaults for minmax are usually in the .2-.3 for min and .8-.9 range for max.
- Parameters
img_shape – Image shape as tuple
minmax – Min and max bbox ratios (as percent of image size)
count – Number of bbox to generate
super_gradients.training.datasets.sg_dataset module
- class super_gradients.training.datasets.sg_dataset.BaseSgVisionDataset(root: str, sample_loader: Callable = <function default_loader>, target_loader: Optional[Callable] = None, collate_fn: Optional[Callable] = None, valid_sample_extensions: tuple = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp'), sample_transform: Optional[Callable] = None, target_transform: Optional[Callable] = None)[source]
Bases:
Generic
[torch.utils.data.dataset.T_co
]- static text_file_loader_func(text_file_path: str, inline_splitter: str = ' ') → list[source]
- text_file_loader_func - Uses a line by line based code to get vectorized data from a text-based file
- param text_file_path
Input text file
- param inline_splitter
The char to use in order to separate between different VALUES of the SAME vector please notice that DIFFERENT VECTORS SHOULD BE IN SEPARATE LINES (’
- ‘) SEPARATED
- return
a list of tuples, where each tuple is a vector of target values
- class super_gradients.training.datasets.sg_dataset.DirectoryDataSet(root: str, samples_sub_directory: str, targets_sub_directory: str, target_extension: str, sample_loader: Callable = <function default_loader>, target_loader: Optional[Callable] = None, collate_fn: Optional[Callable] = None, sample_extensions: tuple = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp'), sample_transform: Optional[Callable] = None, target_transform: Optional[Callable] = None)[source]
Bases:
Generic
[torch.utils.data.dataset.T_co
]- DirectoryDataSet - A PyTorch Vision Data Set extension that receives a root Dir and two separate sub directories:
Sub-Directory for Samples
Sub-Directory for Targets
- class super_gradients.training.datasets.sg_dataset.ListDataset(root, file, sample_loader: Callable = <function default_loader>, target_loader: Optional[Callable] = None, collate_fn: Optional[Callable] = None, sample_extensions: tuple = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp'), sample_transform: Optional[Callable] = None, target_transform: Optional[Callable] = None, target_extension='.npy')[source]
Bases:
Generic
[torch.utils.data.dataset.T_co
]- ListDataset - A PyTorch Vision Data Set extension that receives a file with FULL PATH to each of the samples.
Then, the assumption is that for every sample, there is a * matching target * in the same path but with a different extension, i.e:
- for the samples paths: (That appear in the list file)
/root/dataset/class_x/sample1.png /root/dataset/class_y/sample123.png
- the matching labels paths: (That DO NOT appear in the list file)
/root/dataset/class_x/sample1.ext /root/dataset/class_y/sample123.ext