Struct CompileSpec ¶
-
Defined in File trtorch.h
Nested Relationships ¶
Nested Types ¶
Struct Documentation ¶
-
struct
trtorch
::
CompileSpec
¶
-
Settings data structure for TRTorch compilation
Public Types
Public Functions
-
CompileSpec
( std::vector< InputRange > input_ranges ) ¶
-
Construct a new Extra Info object from input ranges. Each entry in the vector represents a input and should be provided in call order.
Use this constructor if you want to use dynamic shape
- Parameters
-
-
input_ranges
:
-
-
CompileSpec
( std::vector<std::vector<int64_t>> fixed_sizes ) ¶
-
Construct a new Extra Info object Convienence constructor to set fixed input size from vectors describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.
This constructor should be use as a convience in the case that all inputs are static sized and you are okay with default input dtype and formats (FP32 for FP32 and INT8 weights, FP16 for FP16 weights, contiguous)
- Parameters
-
-
fixed_sizes
:
-
-
CompileSpec
( std::vector<c10::ArrayRef<int64_t>> fixed_sizes ) ¶
-
Construct a new Extra Info object Convienence constructor to set fixed input size from c10::ArrayRef’s (the output of tensor.sizes()) describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.
This constructor should be use as a convience in the case that all inputs are static sized and you are okay with default input dtype and formats (FP32 for FP32 and INT8 weights, FP16 for FP16 weights, contiguous)
- Parameters
-
-
fixed_sizes
:
-
Public Members
-
std::vector<
Input
>
inputs
¶
-
Specifications for inputs to the engine, can either be a single size or a range defined by min, opt and max sizes Users can also specify expected input type as well as tensor memory format.
Order in vector should match call order for the function
-
std::vector<
InputRange
>
input_ranges
¶
-
Sizes for inputs to engine, can either be a single size or a range defined by Min, Optimal, Max sizes
Order is should match call order
-
std::set<
DataType
>
enabled_precisions
= { DataType :: kFloat } ¶
-
The set of precisions TensorRT is allowed to use for kernels during compilation.
-
bool
disable_tf32
= false ¶
-
Prevent Float32 layers from using TF32 data format
TF32 computes inner products by rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas. This is the behavior of FP32 layers by default.
-
bool
sparse_weights
= false ¶
-
Enable sparsity for weights of conv and FC layers
-
bool
refit
= false ¶
-
Build a refitable engine
-
bool
debug
= false ¶
-
Build a debugable engine
-
bool
truncate_long_and_double
= false ¶
-
Truncate long/double type to int/float type
-
bool
strict_types
= false ¶
-
Restrict operating type to only the lowest enabled operation precision (enabled_precisions)
-
TorchFallback
torch_fallback
¶
-
Settings related to partial compilation.
-
EngineCapability
capability
= EngineCapability :: kSTANDARD ¶
-
Sets the restrictions for the engine (CUDA Safety)
-
uint64_t
num_min_timing_iters
= 2 ¶
-
Number of minimization timing iterations used to select kernels
-
uint64_t
num_avg_timing_iters
= 1 ¶
-
Number of averaging timing iterations used to select kernels
-
uint64_t
workspace_size
= 0 ¶
-
Maximum size of workspace given to TensorRT
-
uint64_t
max_batch_size
= 0 ¶
-
Maximum batch size (must be >= 1 to be set, 0 means not set)
-
nvinfer1::IInt8Calibrator *
ptq_calibrator
= nullptr ¶
-
Calibration dataloaders for each input for post training quantizatiom
-
class
DataType
-
Supported Data Types that can be used with TensorRT engines
This class is compatable with c10::DataTypes (but will check for TRT support) so there should not be a reason that you need to use this type explictly.
Public Types
-
enum
Value
-
Underlying enum class to support the DataType Class
In the case that you need to use the DataType class itself, interface using this enum vs. normal instatination
ex. trtorch::DataType type = DataType::kFloat ;
Values:
-
enumerator
kFloat
-
FP32.
-
enumerator
kHalf
-
FP16.
-
enumerator
kChar
-
INT8.
-
enumerator
kInt
-
INT.
-
enumerator
kBool
-
Bool.
-
enumerator
kUnknown
-
Sentinel value.
-
enumerator
Public Functions
-
DataType
( ) = default
-
Construct a new Data Type object.
-
DataType
( c10::ScalarType t )
-
Construct a new Data Type object from torch type enums.
- Parameters
-
-
t
:
-
-
operator Value
( ) const
-
Get the enum value of the DataType object.
- Return
-
Value
-
operator bool
( ) = delete
-
constexpr
bool
operator==
( DataType other ) const
-
Comparision operator for DataType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator==
( DataType :: Value other ) const
-
Comparision operator for DataType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
enum
-
struct
Device
¶
-
Public Members
-
DeviceType
device_type
¶
-
Setting data structure for device This struct will hold Target device related parameters such as device_type, gpu_id, dla_core.
-
int64_t
gpu_id
¶
-
int64_t
dla_core
¶
-
bool
allow_gpu_fallback
¶
-
(Only used when targeting DLA (device)) Lets engine run layers on GPU if they are not supported on DLA
-
class
DeviceType
-
Supported Device Types that can be used with TensorRT engines
This class is compatable with c10::DeviceTypes (but will check for TRT support) but the only applicable value is at::kCUDA, which maps to DeviceType::kGPU
To use the DataType class itself, interface using the enum vs. normal instatination
ex. trtorch::DeviceType type = DeviceType::kGPU ;
Public Types
-
enum
Value
-
Underlying enum class to support the DeviceType Class
In the case that you need to use the DeviceType class itself, interface using this enum vs. normal instatination
ex. trtorch::DeviceType type = DeviceType::kGPU ;
Values:
-
enumerator
kGPU
-
Target GPU to run engine.
-
enumerator
kDLA
-
Target DLA to run engine.
-
enumerator
Public Functions
-
DeviceType
( ) = default
-
Construct a new Device Type object.
-
DeviceType
( c10::DeviceType t )
-
Construct a new Device Type object from torch device enums Note: The only valid value is torch::kCUDA (torch::kCPU is not supported)
- Parameters
-
-
t
:
-
-
operator Value
( ) const
-
Get the internal value from the Device object.
- Return
-
Value
-
operator bool
( ) = delete
-
constexpr
bool
operator==
( DeviceType other ) const
-
Comparison operator for DeviceType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator!=
( DeviceType other ) const
-
Comparison operator for DeviceType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
enum
-
DeviceType
-
struct
Input
¶
-
A struct to hold an input range (used by TensorRT Optimization profile)
This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.
Public Functions
-
Input
( std::vector<int64_t> shape , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters
-
-
shape
: Input tensor shape -
dtype
: Expected data type for the input (Defaults to Float32) -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( std::vector<int64_t> shape , DataType dtype , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format.
- Parameters
-
-
shape
: Input tensor shape -
dtype
: Expected data type for the input (Defaults to Float32) -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( c10::ArrayRef<int64_t> shape , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters
-
-
shape
: Input tensor shape -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( c10::ArrayRef<int64_t> shape , DataType dtype , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format.
- Parameters
-
-
shape
: Input tensor shape -
dtype
: Expected data type for the input (Defaults to Float32) -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( std::vector<int64_t> min_shape , std::vector<int64_t> opt_shape , std::vector<int64_t> max_shape , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input Range object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters
-
-
min_shape
: Minimum shape for input tensor -
opt_shape
: Target optimization shape for input tensor -
max_shape
: Maximum acceptible shape for input tensor -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( std::vector<int64_t> min_shape , std::vector<int64_t> opt_shape , std::vector<int64_t> max_shape , DataType dtype , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input spec object for a dynamic input size from vectors for minimum shape, optimal shape, and max shape supported sizes optional arguments allow the user to configure expected input shape tensor format.
- Parameters
-
-
min_shape
: Minimum shape for input tensor -
opt_shape
: Target optimization shape for input tensor -
max_shape
: Maximum acceptible shape for input tensor -
dtype
: Expected data type for the input (Defaults to Float32) -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( c10::ArrayRef<int64_t> min_shape , c10::ArrayRef<int64_t> opt_shape , c10::ArrayRef<int64_t> max_shape , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input Range object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters
-
-
min_shape
: Minimum shape for input tensor -
opt_shape
: Target optimization shape for input tensor -
max_shape
: Maximum acceptible shape for input tensor -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
Input
( c10::ArrayRef<int64_t> min_shape , c10::ArrayRef<int64_t> opt_shape , c10::ArrayRef<int64_t> max_shape , DataType dtype , TensorFormat format = TensorFormat :: kContiguous ) ¶
-
Construct a new Input Range object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.
- Parameters
-
-
min_shape
: Minimum shape for input tensor -
opt_shape
: Target optimization shape for input tensor -
max_shape
: Maximum acceptible shape for input tensor -
dtype
: Expected data type for the input (Defaults to Float32) -
format
: Expected tensor format for the input (Defaults to contiguous)
-
-
bool
get_explicit_set_dtype
( ) ¶
Public Members
-
std::vector<int64_t>
min_shape
¶
-
Minimum acceptable input size into the engine.
-
std::vector<int64_t>
opt_shape
¶
-
Optimal input size into the engine (size optimized for given kernels accept any size in min max range)
-
std::vector<int64_t>
max_shape
¶
-
Maximum acceptable input size into the engine.
-
std::vector<int64_t>
shape
¶
-
Input shape to be fed to TensorRT, in the event of a dynamic shape, -1’s will hold the place of variable dimensions
-
TensorFormat
format
¶
-
Expected tensor format for the input.
-
-
struct
InputRange
¶
-
A struct to hold an input range (used by TensorRT Optimization profile)
This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.
Public Functions
-
InputRange
( std::vector<int64_t> opt ) ¶
-
Construct a new Input Range object for static input size from vector.
- Parameters
-
-
opt
:
-
-
InputRange
( c10::ArrayRef<int64_t> opt ) ¶
-
Construct a new Input Range object static input size from c10::ArrayRef (the type produced by tensor.sizes())
- Parameters
-
-
opt
:
-
-
-
class
TensorFormat
-
Public Types
-
enum
Value
-
Underlying enum class to support the TensorFormat Class
In the case that you need to use the TensorFormat class itself, interface using this enum vs. normal instatination
ex. trtorch::TensorFormat type = TensorFormat::kContiguous ;
Values:
-
enumerator
kContiguous
-
Contiguous / NCHW / Linear.
-
enumerator
kChannelsLast
-
Channel Last / NHWC.
-
enumerator
kUnknown
-
Sentinel value.
-
enumerator
Public Functions
-
TensorFormat
( ) = default
-
Construct a new TensorFormat object.
-
constexpr
TensorFormat
( Value t )
-
TensorFormat constructor from enum.
-
TensorFormat
( at::MemoryFormat t )
-
Construct a new TensorFormat object from torch type enums.
- Parameters
-
-
t
:
-
-
operator Value
( ) const
-
Get the enum value of the TensorFormat object.
- Return
-
Value
-
operator bool
( ) = delete
-
constexpr
bool
operator==
( TensorFormat other ) const
-
Comparision operator for TensorFormat .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator==
( TensorFormat :: Value other ) const
-
Comparision operator for TensorFormat .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator!=
( TensorFormat other ) const
-
Comparision operator for TensorFormat .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator!=
( TensorFormat :: Value other ) const
-
Comparision operator for TensorFormat .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
enum
-
struct
TorchFallback
¶
-
A struct to hold fallback info.
Public Functions
-
TorchFallback
( ) = default ¶
-
Construct a default Torch Fallback object, fallback will be off.
-
TorchFallback
( bool enabled ) ¶
-
Construct from a bool.
-
TorchFallback
( bool enabled , uint64_t min_size ) ¶
-
Constructor for setting min_block_size.
Public Members
-
bool
enabled
= false ¶
-
enable the automatic fallback feature
-
uint64_t
min_block_size
= 1 ¶
-
minimum consecutive operation number that needs to be satisfied to convert to TensorRT
-
std::vector<std::string>
forced_fallback_ops
¶
-
A list of names of operations that will explicitly run in PyTorch.
-
std::vector<std::string>
forced_fallback_modules
¶
-
A list of names of modules that will explicitly run in PyTorch.
-
-