super_gradients.training.datasets.dataset_interfaces package

Submodules

super_gradients.training.datasets.dataset_interfaces.dataset_interface module

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface(dataset_params={}, train_loader=None, val_loader=None, test_loader=None, classes=None)[source]

Bases: object

DatasetInterface - This class manages all of the “communiation” the Model has with the Data Sets

download_from_cloud()[source]
build_data_loaders(batch_size_factor=1, num_workers=8, train_batch_size=None, val_batch_size=None, test_batch_size=None, distributed_sampler: bool = False)[source]

define train, val (and optionally test) loaders. The method deals separately with distributed training and standard (non distributed, or parallel training). In the case of distributed training we need to rely on distributed samplers. :param batch_size_factor: int - factor to multiply the batch size (usually for multi gpu) :param num_workers: int - number of workers (parallel processes) for dataloaders :param train_batch_size: int - batch size for train loader, if None will be taken from dataset_params :param val_batch_size: int - batch size for val loader, if None will be taken from dataset_params :param distributed_sampler: boolean flag for distributed training mode :return: train_loader, val_loader, classes: list of classes

get_data_loaders(**kwargs)[source]

Get self.train_loader, self.test_loader, self.classes.

If the data loaders haven’t been initialized yet, build them first.

Parameters

kwargs – kwargs are passed to build_data_loaders.

get_val_sample(num_samples=1)[source]
get_dataset_params()[source]
print_dataset_details()[source]
class super_gradients.training.datasets.dataset_interfaces.dataset_interface.ExternalDatasetInterface(train_loader, val_loader, num_classes, dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

get_data_loaders(batch_size_factor: int = 1, num_workers: int = 8, train_batch_size: Optional[int] = None, val_batch_size: Optional[int] = None, distributed_sampler: bool = False)[source]

Get self.train_loader, self.test_loader, self.classes.

If the data loaders haven’t been initialized yet, build them first.

Parameters

kwargs – kwargs are passed to build_data_loaders.

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.LibraryDatasetInterface(name='cifar10', dataset_params={}, to_cutout=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.Cifar10DatasetInterface(dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.LibraryDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.Cifar100DatasetInterface(dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.LibraryDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface(trainset, dataset_params={}, classes=None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

get_data_loaders(batch_size_factor=1, num_workers=8, train_batch_size=None, val_batch_size=None, distributed_sampler=False)[source]

Get self.train_loader, self.test_loader, self.classes.

If the data loaders haven’t been initialized yet, build them first.

Parameters

kwargs – kwargs are passed to build_data_loaders.

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.ClassificationTestDatasetInterface(dataset_params={}, image_size=32, batch_size=5, classes=None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.SegmentationTestDatasetInterface(dataset_params={}, image_size=512, batch_size=4)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.DetectionTestDatasetInterface(dataset_params={}, image_size=320, batch_size=4)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestYoloDetectionDatasetInterface(dataset_params={}, input_dims=(3, 32, 32), batch_size=5)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

note: the output size is (batch_size, 6) in the test while in real training the size of axis 0 can vary (the number of bounding boxes)

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.ImageNetDatasetInterface(dataset_params={}, data_dir='/data/Imagenet')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.TinyImageNetDatasetInterface(dataset_params={}, data_dir='/data/TinyImagenet')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.ClassificationDatasetInterface(normalization_mean=(0, 0, 0), normalization_std=(1, 1, 1), resolution=64, dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.PascalVOC2012DetectionDataSetInterface(dataset_params=None, cache_labels=False, cache_images=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.PascalVOC2012SegmentationDataSetInterface(dataset_params=None, cache_labels=False, cache_images=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.PascalAUG2012SegmentationDataSetInterface(dataset_params=None, cache_labels=False, cache_images=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDataSetInterfaceBase(dataset_params=None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDetectionDatasetInterface(dataset_params=None, cache_labels=False, cache_images=False, train_list_file='train2017.txt', val_list_file='val2017.txt')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDataSetInterfaceBase

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoSegmentationDatasetInterface(dataset_params=None, cache_labels: bool = False, cache_images: bool = False, dataset_classes_inclusion_tuples_list: Optional[list] = None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDataSetInterfaceBase

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCo2014DetectionDatasetInterface(dataset_params=None, cache_labels=False, cache_images=False, train_list_file='train2014.txt', val_list_file='val2014.txt')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDetectionDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.dataset_interface.CityscapesDatasetInterface(dataset_params=None, cache_labels: bool = False, cache_images: bool = False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

Module contents

class super_gradients.training.datasets.dataset_interfaces.DatasetInterface(dataset_params={}, train_loader=None, val_loader=None, test_loader=None, classes=None)[source]

Bases: object

DatasetInterface - This class manages all of the “communiation” the Model has with the Data Sets

download_from_cloud()[source]
build_data_loaders(batch_size_factor=1, num_workers=8, train_batch_size=None, val_batch_size=None, test_batch_size=None, distributed_sampler: bool = False)[source]

define train, val (and optionally test) loaders. The method deals separately with distributed training and standard (non distributed, or parallel training). In the case of distributed training we need to rely on distributed samplers. :param batch_size_factor: int - factor to multiply the batch size (usually for multi gpu) :param num_workers: int - number of workers (parallel processes) for dataloaders :param train_batch_size: int - batch size for train loader, if None will be taken from dataset_params :param val_batch_size: int - batch size for val loader, if None will be taken from dataset_params :param distributed_sampler: boolean flag for distributed training mode :return: train_loader, val_loader, classes: list of classes

get_data_loaders(**kwargs)[source]

Get self.train_loader, self.test_loader, self.classes.

If the data loaders haven’t been initialized yet, build them first.

Parameters

kwargs – kwargs are passed to build_data_loaders.

get_val_sample(num_samples=1)[source]
get_dataset_params()[source]
print_dataset_details()[source]
class super_gradients.training.datasets.dataset_interfaces.TestDatasetInterface(trainset, dataset_params={}, classes=None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

get_data_loaders(batch_size_factor=1, num_workers=8, train_batch_size=None, val_batch_size=None, distributed_sampler=False)[source]

Get self.train_loader, self.test_loader, self.classes.

If the data loaders haven’t been initialized yet, build them first.

Parameters

kwargs – kwargs are passed to build_data_loaders.

class super_gradients.training.datasets.dataset_interfaces.LibraryDatasetInterface(name='cifar10', dataset_params={}, to_cutout=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.ClassificationDatasetInterface(normalization_mean=(0, 0, 0), normalization_std=(1, 1, 1), resolution=64, dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.Cifar10DatasetInterface(dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.LibraryDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.Cifar100DatasetInterface(dataset_params={})[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.LibraryDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.ImageNetDatasetInterface(dataset_params={}, data_dir='/data/Imagenet')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.TinyImageNetDatasetInterface(dataset_params={}, data_dir='/data/TinyImagenet')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.CoCoDetectionDatasetInterface(dataset_params=None, cache_labels=False, cache_images=False, train_list_file='train2017.txt', val_list_file='val2017.txt')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDataSetInterfaceBase

class super_gradients.training.datasets.dataset_interfaces.CoCo2014DetectionDatasetInterface(dataset_params=None, cache_labels=False, cache_images=False, train_list_file='train2014.txt', val_list_file='val2014.txt')[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDetectionDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.CoCoSegmentationDatasetInterface(dataset_params=None, cache_labels: bool = False, cache_images: bool = False, dataset_classes_inclusion_tuples_list: Optional[list] = None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.CoCoDataSetInterfaceBase

class super_gradients.training.datasets.dataset_interfaces.PascalAUG2012SegmentationDataSetInterface(dataset_params=None, cache_labels=False, cache_images=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.PascalVOC2012SegmentationDataSetInterface(dataset_params=None, cache_labels=False, cache_images=False)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

class super_gradients.training.datasets.dataset_interfaces.TestYoloDetectionDatasetInterface(dataset_params={}, input_dims=(3, 32, 32), batch_size=5)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.DatasetInterface

note: the output size is (batch_size, 6) in the test while in real training the size of axis 0 can vary (the number of bounding boxes)

class super_gradients.training.datasets.dataset_interfaces.SegmentationTestDatasetInterface(dataset_params={}, image_size=512, batch_size=4)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.DetectionTestDatasetInterface(dataset_params={}, image_size=320, batch_size=4)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface

class super_gradients.training.datasets.dataset_interfaces.ClassificationTestDatasetInterface(dataset_params={}, image_size=32, batch_size=5, classes=None)[source]

Bases: super_gradients.training.datasets.dataset_interfaces.dataset_interface.TestDatasetInterface