Struct CompileSpec

Struct Documentation

struct torch_tensorrt::torchscript :: CompileSpec

Settings data structure for Torch-TensorRT TorchScript compilation

Public Functions

CompileSpec ( std::vector<std::vector<int64_t>> fixed_sizes )

Construct a new Compile Spec object Convienence constructor to set fixed input size from vectors describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.

This constructor should be use as a convience in the case that all inputs are static sized and you are okay with default input dtype and formats (FP32 for FP32 and INT8 weights, FP16 for FP16 weights, contiguous)

Parameters
  • fixed_sizes :

CompileSpec ( std::vector<c10::ArrayRef<int64_t>> fixed_sizes )

Construct a new Extra Info object Convienence constructor to set fixed input size from c10::ArrayRef’s (the output of tensor.sizes()) describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.

This constructor should be use as a convience in the case that all inputs are static sized and you are okay with default input dtype and formats (FP32 for FP32 and INT8 weights, FP16 for FP16 weights, contiguous)

Parameters
  • fixed_sizes :

CompileSpec ( std::vector< Input > inputs )

Construct a new Extra Info object from input ranges. Each entry in the vector represents a input and should be provided in call order.

Use this constructor to define inputs with dynamic shape, specific input types or tensor formats

Parameters
  • inputs :

Public Members

std::vector< Input > inputs

Specifications for inputs to the engine, can either be a single size or a range defined by min, opt and max sizes Users can also specify expected input type as well as tensor memory format.

Order in vector should match call order for the function

std::set< DataType > enabled_precisions = { DataType :: kFloat }

The set of precisions TensorRT is allowed to use for kernels during compilation.

bool disable_tf32 = false

Prevent Float32 layers from using TF32 data format

TF32 computes inner products by rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas. This is the behavior of FP32 layers by default.

bool sparse_weights = false

Enable sparsity for weights of conv and FC layers

bool refit = false

Build a refitable engine

bool debug = false

Build a debugable engine

bool truncate_long_and_double = false

Truncate long/double type to int/float type

Device device

Target Device

EngineCapability capability = EngineCapability :: kSTANDARD

Sets the restrictions for the engine (CUDA Safety)

uint64_t num_min_timing_iters = 2

Number of minimization timing iterations used to select kernels

uint64_t num_avg_timing_iters = 1

Number of averaging timing iterations used to select kernels

uint64_t workspace_size = 0

Maximum size of workspace given to TensorRT

nvinfer1::IInt8Calibrator * ptq_calibrator = nullptr

Calibration dataloaders for each input for post training quantizatiom

bool require_full_compilation = false

Require the full module be compiled to TensorRT instead of potentially running unsupported operations in PyTorch

uint64_t min_block_size = 3

Minimum number of contiguous supported operators to compile a subgraph to TensorRT

std::vector<std::string> torch_executed_ops

List of aten operators that must be run in PyTorch. An error will be thrown if this list is not empty but require_full_compilation is True

std::vector<std::string> torch_executed_modules

List of modules that must be run in PyTorch. An error will be thrown if this list is not empty but require_full_compilation is True