Struct CompileSpec

Nested Relationships

Nested Types

Struct Documentation

struct trtorch :: CompileSpec

Settings data structure for TRTorch compilation

Public Types

enum EngineCapability

Emum for selecting engine capability

Values:

enumerator kDEFAULT
enumerator kSAFE_GPU
enumerator kSAFE_DLA

Public Functions

CompileSpec ( std::vector< InputRange > input_ranges )

Construct a new Extra Info object from input ranges. Each entry in the vector represents a input and should be provided in call order.

Use this constructor if you want to use dynamic shape

Parameters
  • input_ranges :

CompileSpec ( std::vector<std::vector<int64_t>> fixed_sizes )

Construct a new Extra Info object Convienence constructor to set fixed input size from vectors describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.

Parameters
  • fixed_sizes :

CompileSpec ( std::vector<c10::ArrayRef<int64_t>> fixed_sizes )

Construct a new Extra Info object Convienence constructor to set fixed input size from c10::ArrayRef’s (the output of tensor.sizes()) describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.

Parameters
  • fixed_sizes :

Public Members

std::vector< InputRange > input_ranges

Sizes for inputs to engine, can either be a single size or a range defined by Min, Optimal, Max sizes

Order is should match call order

DataType op_precision = DataType :: kFloat

Default operating precision for the engine

bool refit = false

Build a refitable engine

bool debug = false

Build a debugable engine

bool strict_types = false

Restrict operating type to only set default operation precision (op_precision)

bool allow_gpu_fallback = true

(Only used when targeting DLA (device)) Lets engine run layers on GPU if they are not supported on DLA

DeviceType device = DeviceType :: kGPU

Target device type

EngineCapability capability = EngineCapability :: kDEFAULT

Sets the restrictions for the engine (CUDA Safety)

uint64_t num_min_timing_iters = 2

Number of minimization timing iterations used to select kernels

uint64_t num_avg_timing_iters = 1

Number of averaging timing iterations used to select kernels

uint64_t workspace_size = 0

Maximum size of workspace given to TensorRT

uint64_t max_batch_size = 0

Maximum batch size (must be >= 1 to be set, 0 means not set)

nvinfer1::IInt8Calibrator * ptq_calibrator = nullptr

Calibration dataloaders for each input for post training quantizatiom

class DataType

Supported Data Types that can be used with TensorRT engines

This class is compatable with c10::DataTypes (but will check for TRT support) so there should not be a reason that you need to use this type explictly.

Public Types

enum Value

Underlying enum class to support the DataType Class

In the case that you need to use the DataType class itself, interface using this enum vs. normal instatination

ex. trtorch::DataType type = DataType::kFloat ;

Values:

enumerator kFloat

FP32.

enumerator kHalf

FP16.

enumerator kChar

INT8.

Public Functions

DataType ( ) = default

Construct a new Data Type object.

constexpr DataType ( Value t )

DataType constructor from enum.

DataType ( c10::ScalarType t )

Construct a new Data Type object from torch type enums.

Parameters
  • t :

operator Value ( ) const

Get the enum value of the DataType object.

Return

Value

operator bool ( ) = delete
constexpr bool operator== ( DataType other ) const

Comparision operator for DataType .

Return

true

Return

false

Parameters
  • other :

constexpr bool operator== ( DataType :: Value other ) const

Comparision operator for DataType .

Return

true

Return

false

Parameters
  • other :

constexpr bool operator!= ( DataType other ) const

Comparision operator for DataType .

Return

true

Return

false

Parameters
  • other :

constexpr bool operator!= ( DataType :: Value other ) const

Comparision operator for DataType .

Return

true

Return

false

Parameters
  • other :

class DeviceType

Supported Device Types that can be used with TensorRT engines

This class is compatable with c10::DeviceTypes (but will check for TRT support) but the only applicable value is at::kCUDA, which maps to DeviceType::kGPU

To use the DataType class itself, interface using the enum vs. normal instatination

ex. trtorch::DeviceType type = DeviceType::kGPU ;

Public Types

enum Value

Underlying enum class to support the DeviceType Class

In the case that you need to use the DeviceType class itself, interface using this enum vs. normal instatination

ex. trtorch::DeviceType type = DeviceType::kGPU ;

Values:

enumerator kGPU

Target GPU to run engine.

enumerator kDLA

Target DLA to run engine.

Public Functions

DeviceType ( ) = default

Construct a new Device Type object.

constexpr DeviceType ( Value t )

Construct a new Device Type object from internal enum.

DeviceType ( c10::DeviceType t )

Construct a new Device Type object from torch device enums Note: The only valid value is torch::kCUDA (torch::kCPU is not supported)

Parameters
  • t :

operator Value ( ) const

Get the internal value from the Device object.

Return

Value

operator bool ( ) = delete
constexpr bool operator== ( DeviceType other ) const

Comparison operator for DeviceType .

Return

true

Return

false

Parameters
  • other :

constexpr bool operator!= ( DeviceType other ) const

Comparison operator for DeviceType .

Return

true

Return

false

Parameters
  • other :

struct InputRange

A struct to hold an input range (used by TensorRT Optimization profile)

This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.

Public Functions

InputRange ( std::vector<int64_t> opt )

Construct a new Input Range object for static input size from vector.

Parameters
  • opt :

InputRange ( c10::ArrayRef<int64_t> opt )

Construct a new Input Range object static input size from c10::ArrayRef (the type produced by tensor.sizes())

Parameters
  • opt :

InputRange ( std::vector<int64_t> min , std::vector<int64_t> opt , std::vector<int64_t> max )

Construct a new Input Range object dynamic input size from vectors for min, opt, and max supported sizes.

Parameters
  • min :

  • opt :

  • max :

InputRange ( c10::ArrayRef<int64_t> min , c10::ArrayRef<int64_t> opt , c10::ArrayRef<int64_t> max )

Construct a new Input Range object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.

Parameters
  • min :

  • opt :

  • max :

Public Members

std::vector<int64_t> min

Minimum acceptable input size into the engine.

std::vector<int64_t> opt

Optimal input size into the engine (gets best performace)

std::vector<int64_t> max

Maximum acceptable input size into the engine.