Struct CompileSpec ¶
-
Defined in File trtorch.h
Nested Relationships ¶
Nested Types ¶
Struct Documentation ¶
-
struct
trtorch
::
CompileSpec
¶
-
Settings data structure for TRTorch compilation
Public Types
Public Functions
-
CompileSpec
( std::vector< InputRange > input_ranges ) ¶
-
Construct a new Extra Info object from input ranges. Each entry in the vector represents a input and should be provided in call order.
Use this constructor if you want to use dynamic shape
- Parameters
-
-
input_ranges
:
-
-
CompileSpec
( std::vector<std::vector<int64_t>> fixed_sizes ) ¶
-
Construct a new Extra Info object Convienence constructor to set fixed input size from vectors describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.
- Parameters
-
-
fixed_sizes
:
-
-
CompileSpec
( std::vector<c10::ArrayRef<int64_t>> fixed_sizes ) ¶
-
Construct a new Extra Info object Convienence constructor to set fixed input size from c10::ArrayRef’s (the output of tensor.sizes()) describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.
- Parameters
-
-
fixed_sizes
:
-
Public Members
-
std::vector<
InputRange
>
input_ranges
¶
-
Sizes for inputs to engine, can either be a single size or a range defined by Min, Optimal, Max sizes
Order is should match call order
-
bool
refit
= false ¶
-
Build a refitable engine
-
bool
debug
= false ¶
-
Build a debugable engine
-
bool
strict_types
= false ¶
-
Restrict operating type to only set default operation precision (op_precision)
-
bool
allow_gpu_fallback
= true ¶
-
(Only used when targeting DLA (device)) Lets engine run layers on GPU if they are not supported on DLA
-
DeviceType
device
= DeviceType ::kGPU ¶
-
Target device type
-
EngineCapability
capability
= EngineCapability :: kDEFAULT ¶
-
Sets the restrictions for the engine (CUDA Safety)
-
uint64_t
num_min_timing_iters
= 2 ¶
-
Number of minimization timing iterations used to select kernels
-
uint64_t
num_avg_timing_iters
= 1 ¶
-
Number of averaging timing iterations used to select kernels
-
uint64_t
workspace_size
= 0 ¶
-
Maximum size of workspace given to TensorRT
-
uint64_t
max_batch_size
= 0 ¶
-
Maximum batch size (must be >= 1 to be set, 0 means not set)
-
nvinfer1::IInt8Calibrator *
ptq_calibrator
= nullptr ¶
-
Calibration dataloaders for each input for post training quantizatiom
-
class
DataType
-
Supported Data Types that can be used with TensorRT engines
This class is compatable with c10::DataTypes (but will check for TRT support) so there should not be a reason that you need to use this type explictly.
Public Types
-
enum
Value
-
Underlying enum class to support the DataType Class
In the case that you need to use the DataType class itself, interface using this enum vs. normal instatination
ex. trtorch::DataType type = DataType::kFloat ;
Values:
-
enumerator
kFloat
-
FP32.
-
enumerator
kHalf
-
FP16.
-
enumerator
kChar
-
INT8.
-
enumerator
Public Functions
-
DataType
( ) = default
-
Construct a new Data Type object.
-
DataType
( c10::ScalarType t )
-
Construct a new Data Type object from torch type enums.
- Parameters
-
-
t
:
-
-
operator Value
( ) const
-
Get the enum value of the DataType object.
- Return
-
Value
-
operator bool
( ) = delete
-
constexpr
bool
operator==
( DataType other ) const
-
Comparision operator for DataType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator==
( DataType :: Value other ) const
-
Comparision operator for DataType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
enum
-
class
DeviceType
-
Supported Device Types that can be used with TensorRT engines
This class is compatable with c10::DeviceTypes (but will check for TRT support) but the only applicable value is at::kCUDA, which maps to DeviceType::kGPU
To use the DataType class itself, interface using the enum vs. normal instatination
ex. trtorch::DeviceType type = DeviceType::kGPU ;
Public Types
-
enum
Value
-
Underlying enum class to support the DeviceType Class
In the case that you need to use the DeviceType class itself, interface using this enum vs. normal instatination
ex. trtorch::DeviceType type = DeviceType::kGPU ;
Values:
-
enumerator
kGPU
-
Target GPU to run engine.
-
enumerator
kDLA
-
Target DLA to run engine.
-
enumerator
Public Functions
-
DeviceType
( ) = default
-
Construct a new Device Type object.
-
constexpr
DeviceType
( Value t )
-
Construct a new Device Type object from internal enum.
-
DeviceType
( c10::DeviceType t )
-
Construct a new Device Type object from torch device enums Note: The only valid value is torch::kCUDA (torch::kCPU is not supported)
- Parameters
-
-
t
:
-
-
operator Value
( ) const
-
Get the internal value from the Device object.
- Return
-
Value
-
operator bool
( ) = delete
-
constexpr
bool
operator==
( DeviceType other ) const
-
Comparison operator for DeviceType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
constexpr
bool
operator!=
( DeviceType other ) const
-
Comparison operator for DeviceType .
- Return
-
true
- Return
-
false
- Parameters
-
-
other
:
-
-
enum
-
struct
InputRange
¶
-
A struct to hold an input range (used by TensorRT Optimization profile)
This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.
Public Functions
-
InputRange
( std::vector<int64_t> opt ) ¶
-
Construct a new Input Range object for static input size from vector.
- Parameters
-
-
opt
:
-
-
InputRange
( c10::ArrayRef<int64_t> opt ) ¶
-
Construct a new Input Range object static input size from c10::ArrayRef (the type produced by tensor.sizes())
- Parameters
-
-
opt
:
-
-
InputRange
( std::vector<int64_t> min , std::vector<int64_t> opt , std::vector<int64_t> max ) ¶
-
Construct a new Input Range object dynamic input size from vectors for min, opt, and max supported sizes.
- Parameters
-
-
min
: -
opt
: -
max
:
-
-
InputRange
( c10::ArrayRef<int64_t> min , c10::ArrayRef<int64_t> opt , c10::ArrayRef<int64_t> max ) ¶
-
Construct a new Input Range object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.
- Parameters
-
-
min
: -
opt
: -
max
:
-
-
-