Deploying TRTorch Programs

After compiling and saving TRTorch programs there is no longer a strict dependency on the full TRTorch library. All that is required to run a compiled program is the runtime. There are therfore a couple options to deploy your programs other than shipping the full trtorch compiler with your applications.

TRTorch package / libtrtorch.so

Once a program is compiled, you run it using the standard PyTorch APIs. All that is required is that the package must be imported in python or linked in C++.

Runtime Library

Distributed with the C++ distribution is libtrtorchrt.so . This library only contains the components necessary to run TRTorch programs. Instead of linking libtrtorch.so or importing trtorch you can link libtrtorchrt.so in your deployment programs or use DL_OPEN or LD_PRELOAD . For python you can load the runtime with torch.ops.load_library("libtrtorchrt.so") . You can then continue to use programs just as you would otherwise via PyTorch API.

Note

If you are using the standard distribution of PyTorch in Python on x86, likely you will need the pre-cxx11-abi variant of libtrtorchrt.so , check Installation documentation for more details.