[1]:
# Copyright 2019 NVIDIA Corporation. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
Torch-TensorRT Getting Started - CitriNet ¶
Overview ¶
Citrinet is an acoustic model used for the speech to text recognition task. It is a version of QuartzNet that extends ContextNet , utilizing subword encoding (via Word Piece tokenization) and Squeeze-and-Excitation(SE) mechanism and are therefore smaller than QuartzNet models.
CitriNet models take in audio segments and transcribe them to letter, byte pair, or word piece sequences.
Learning objectives ¶
This notebook demonstrates the steps for optimizing a pretrained CitriNet model with Torch-TensorRT, and running it to test the speedup obtained.
Content ¶
## 1. Requirements
Follow the steps in
README
to prepare a Docker container, within which you can run this notebook. This notebook assumes that you are within a Jupyter environment in a docker container with Torch-TensorRT installed, such as an NGC monthly release of
nvcr.io/nvidia/pytorch:<yy.mm>-py3
(where
yy
indicates the last two numbers of a calendar year, and
mm
indicates the month in two-digit numerical form)
Now that you are in the docker, the next step is to install the required dependencies.
[2]:
# Install dependencies
!pip install wget
!apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y libsndfile1 ffmpeg
!pip install Cython
## Install NeMo
!pip install nemo_toolkit[all]==1.5.1
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: wget in /opt/conda/lib/python3.8/site-packages (3.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
libsndfile1 is already the newest version (1.0.28-7ubuntu0.1).
ffmpeg is already the newest version (7:4.2.4-1ubuntu0.1).
0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: Cython in /opt/conda/lib/python3.8/site-packages (0.29.28)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: nemo_toolkit[all]==1.5.1 in /opt/conda/lib/python3.8/site-packages (1.5.1)
Requirement already satisfied: numpy>=1.18.2 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.22.3)
Requirement already satisfied: onnx>=1.7.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.10.1)
Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.8.2)
Requirement already satisfied: tqdm>=4.41.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.63.0)
Requirement already satisfied: sentencepiece<1.0.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.1.96)
Requirement already satisfied: wget in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (3.2)
Requirement already satisfied: numba in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.53.1)
Requirement already satisfied: torch in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.12.0a0+2c916ef)
Requirement already satisfied: unidecode in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.3.4)
Requirement already satisfied: frozendict in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.3.2)
Requirement already satisfied: wrapt in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.14.0)
Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.24.2)
Requirement already satisfied: ruamel.yaml in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.17.21)
Requirement already satisfied: pesq in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.0.3)
Requirement already satisfied: torchvision in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.13.0a0)
Requirement already satisfied: gdown in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.4.0)
Requirement already satisfied: editdistance in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.6.0)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.21.45)
Requirement already satisfied: isort[requirements]<5 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.3.21)
Requirement already satisfied: hydra-core>=1.1.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.1.2)
Requirement already satisfied: youtokentome>=1.0.5 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.0.6)
Requirement already satisfied: pytorch-lightning>=1.5.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.6.1)
Requirement already satisfied: jieba in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.42.1)
Requirement already satisfied: fasttext in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.9.2)
Requirement already satisfied: soundfile in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.10.3.post1)
Requirement already satisfied: kaldiio in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.17.2)
Requirement already satisfied: pangu in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.0.6.1)
Requirement already satisfied: kaldi-python-io in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.2.2)
Requirement already satisfied: parameterized in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.8.1)
Requirement already satisfied: h5py in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (3.6.0)
Requirement already satisfied: rapidfuzz in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.0.10)
Requirement already satisfied: marshmallow in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (3.15.0)
Requirement already satisfied: opencc in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.1.3)
Requirement already satisfied: braceexpand in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.1.7)
Requirement already satisfied: omegaconf>=2.1.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.1.2)
Requirement already satisfied: sphinx in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.4.0)
Requirement already satisfied: pillow in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (9.0.0)
Requirement already satisfied: wordninja==2.0.0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.0.0)
Requirement already satisfied: torch-stft in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.1.4)
Requirement already satisfied: sox in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.4.1)
Requirement already satisfied: librosa in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.9.1)
Requirement already satisfied: regex in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2022.3.15)
Requirement already satisfied: sacrebleu[ja] in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.0.0)
Requirement already satisfied: black==19.10b0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (19.10b0)
Requirement already satisfied: pydub in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.25.1)
Requirement already satisfied: sphinxcontrib-bibtex in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.4.2)
Requirement already satisfied: inflect in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (5.5.2)
Requirement already satisfied: pyannote.core in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.4)
Requirement already satisfied: packaging in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (21.3)
Requirement already satisfied: kaldi-io in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.9.4)
Requirement already satisfied: pyannote.metrics in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (3.2)
Requirement already satisfied: g2p-en in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.1.0)
Requirement already satisfied: matplotlib in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (3.5.1)
Requirement already satisfied: torchmetrics>=0.4.1rc0 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.8.0)
Requirement already satisfied: nltk>=3.6.5 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (3.7)
Requirement already satisfied: pyyaml<6 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (5.4.1)
Requirement already satisfied: scipy in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.6.3)
Requirement already satisfied: ipywidgets in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (7.7.0)
Requirement already satisfied: pytest in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (6.2.5)
Requirement already satisfied: pandas in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (1.3.5)
Requirement already satisfied: pytest-runner in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (6.0.0)
Requirement already satisfied: transformers>=4.0.1 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (4.18.0)
Requirement already satisfied: sacremoses>=0.0.43 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.0.49)
Requirement already satisfied: pystoi in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.3.3)
Requirement already satisfied: attrdict in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (2.0.1)
Requirement already satisfied: webdataset<=0.1.62,>=0.1.48 in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.1.62)
Requirement already satisfied: wandb in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.12.15)
Requirement already satisfied: pypinyin in /opt/conda/lib/python3.8/site-packages (from nemo_toolkit[all]==1.5.1) (0.46.0)
Requirement already satisfied: attrs>=18.1.0 in /opt/conda/lib/python3.8/site-packages (from black==19.10b0->nemo_toolkit[all]==1.5.1) (21.4.0)
Requirement already satisfied: appdirs in /opt/conda/lib/python3.8/site-packages (from black==19.10b0->nemo_toolkit[all]==1.5.1) (1.4.4)
Requirement already satisfied: typed-ast>=1.4.0 in /opt/conda/lib/python3.8/site-packages (from black==19.10b0->nemo_toolkit[all]==1.5.1) (1.5.3)
Requirement already satisfied: pathspec<1,>=0.6 in /opt/conda/lib/python3.8/site-packages (from black==19.10b0->nemo_toolkit[all]==1.5.1) (0.9.0)
Requirement already satisfied: click>=6.5 in /opt/conda/lib/python3.8/site-packages (from black==19.10b0->nemo_toolkit[all]==1.5.1) (8.0.4)
Requirement already satisfied: toml>=0.9.4 in /opt/conda/lib/python3.8/site-packages (from black==19.10b0->nemo_toolkit[all]==1.5.1) (0.10.2)
Requirement already satisfied: antlr4-python3-runtime==4.8 in /opt/conda/lib/python3.8/site-packages (from hydra-core>=1.1.0->nemo_toolkit[all]==1.5.1) (4.8)
Requirement already satisfied: importlib-resources<5.3 in /opt/conda/lib/python3.8/site-packages (from hydra-core>=1.1.0->nemo_toolkit[all]==1.5.1) (5.2.3)
Requirement already satisfied: zipp>=3.1.0 in /opt/conda/lib/python3.8/site-packages (from importlib-resources<5.3->hydra-core>=1.1.0->nemo_toolkit[all]==1.5.1) (3.7.0)
Requirement already satisfied: pip-api in /opt/conda/lib/python3.8/site-packages (from isort[requirements]<5->nemo_toolkit[all]==1.5.1) (0.0.29)
Requirement already satisfied: pipreqs in /opt/conda/lib/python3.8/site-packages (from isort[requirements]<5->nemo_toolkit[all]==1.5.1) (0.4.11)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.8/site-packages (from matplotlib->nemo_toolkit[all]==1.5.1) (4.31.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib->nemo_toolkit[all]==1.5.1) (1.4.0)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib->nemo_toolkit[all]==1.5.1) (3.0.7)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.8/site-packages (from matplotlib->nemo_toolkit[all]==1.5.1) (0.11.0)
Requirement already satisfied: joblib in /opt/conda/lib/python3.8/site-packages (from nltk>=3.6.5->nemo_toolkit[all]==1.5.1) (1.1.0)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/conda/lib/python3.8/site-packages (from onnx>=1.7.0->nemo_toolkit[all]==1.5.1) (4.1.1)
Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from onnx>=1.7.0->nemo_toolkit[all]==1.5.1) (1.16.0)
Requirement already satisfied: protobuf>=3.12.2 in /opt/conda/lib/python3.8/site-packages (from onnx>=1.7.0->nemo_toolkit[all]==1.5.1) (3.19.4)
Requirement already satisfied: pyDeprecate<0.4.0,>=0.3.1 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (0.3.2)
Requirement already satisfied: tensorboard>=2.2.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2.8.0)
Requirement already satisfied: fsspec[http]!=2021.06.0,>=2021.05.0 in /opt/conda/lib/python3.8/site-packages (from pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2022.2.0)
Requirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2.27.1)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.8/site-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (3.8.1)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2.0.3)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (3.3.6)
Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (59.5.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (0.4.6)
Requirement already satisfied: google-auth<3,>=1.6.3 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2.6.2)
Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (0.37.1)
Requirement already satisfied: grpcio>=1.24.3 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.44.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (0.6.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.8.1)
Requirement already satisfied: absl-py>=0.4 in /opt/conda/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.0.0)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (5.0.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (4.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /opt/conda/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (4.11.3)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (0.4.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.8/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (3.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.26.8)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (3.2.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /opt/conda/lib/python3.8/site-packages (from transformers>=4.0.1->nemo_toolkit[all]==1.5.1) (0.5.1)
Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in /opt/conda/lib/python3.8/site-packages (from transformers>=4.0.1->nemo_toolkit[all]==1.5.1) (0.12.1)
Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from transformers>=4.0.1->nemo_toolkit[all]==1.5.1) (3.6.0)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.3.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.7.2)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (4.0.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (6.0.2)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.2.0)
Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from boto3->nemo_toolkit[all]==1.5.1) (0.5.2)
Requirement already satisfied: botocore<1.25.0,>=1.24.45 in /opt/conda/lib/python3.8/site-packages (from boto3->nemo_toolkit[all]==1.5.1) (1.24.45)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.8/site-packages (from boto3->nemo_toolkit[all]==1.5.1) (1.0.0)
Requirement already satisfied: pybind11>=2.2 in /opt/conda/lib/python3.8/site-packages (from fasttext->nemo_toolkit[all]==1.5.1) (2.9.1)
Requirement already satisfied: distance>=0.1.3 in /opt/conda/lib/python3.8/site-packages (from g2p-en->nemo_toolkit[all]==1.5.1) (0.1.3)
Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.8/site-packages (from gdown->nemo_toolkit[all]==1.5.1) (4.10.0)
Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.8/site-packages (from beautifulsoup4->gdown->nemo_toolkit[all]==1.5.1) (2.3.1)
Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (0.2.0)
Requirement already satisfied: ipython>=4.0.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (8.1.1)
Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (6.9.2)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (1.1.0)
Requirement already satisfied: widgetsnbextension~=3.6.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (3.6.0)
Requirement already satisfied: traitlets>=4.3.1 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (5.1.1)
Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.8/site-packages (from ipywidgets->nemo_toolkit[all]==1.5.1) (5.2.0)
Requirement already satisfied: jupyter-client<8.0 in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (7.1.2)
Requirement already satisfied: psutil in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (5.9.0)
Requirement already satisfied: tornado<7.0,>=4.2 in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (6.1)
Requirement already satisfied: debugpy<2.0,>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (1.5.1)
Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (1.5.4)
Requirement already satisfied: matplotlib-inline<0.2.0,>=0.1.0 in /opt/conda/lib/python3.8/site-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (0.1.3)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.7.5)
Requirement already satisfied: decorator in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (5.1.1)
Requirement already satisfied: pygments>=2.4.0 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (2.11.2)
Requirement already satisfied: stack-data in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.2.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (3.0.27)
Requirement already satisfied: backcall in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.2.0)
Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.18.1)
Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.8/site-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (4.8.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.8.3)
Requirement already satisfied: entrypoints in /opt/conda/lib/python3.8/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (0.4)
Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.8/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (22.3.0)
Requirement already satisfied: jupyter-core>=4.6.0 in /opt/conda/lib/python3.8/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets->nemo_toolkit[all]==1.5.1) (4.9.2)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.8/site-packages (from nbformat>=4.2.0->ipywidgets->nemo_toolkit[all]==1.5.1) (4.4.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.18.1)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.8/site-packages (from pexpect>4.3->ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.2.5)
Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.8/site-packages (from widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (6.4.1)
Requirement already satisfied: Send2Trash>=1.5.0 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (1.8.0)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.13.1)
Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.13.3)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (3.0.3)
Requirement already satisfied: nbconvert in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (6.4.4)
Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.8/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (21.3.0)
Requirement already satisfied: argon2-cffi-bindings in /opt/conda/lib/python3.8/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (21.2.0)
Requirement already satisfied: cffi>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (1.15.0)
Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (2.21)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (2.1.1)
Requirement already satisfied: resampy>=0.2.2 in /opt/conda/lib/python3.8/site-packages (from librosa->nemo_toolkit[all]==1.5.1) (0.2.2)
Requirement already satisfied: pooch>=1.0 in /opt/conda/lib/python3.8/site-packages (from librosa->nemo_toolkit[all]==1.5.1) (1.6.0)
Requirement already satisfied: audioread>=2.1.5 in /opt/conda/lib/python3.8/site-packages (from librosa->nemo_toolkit[all]==1.5.1) (2.1.9)
Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in /opt/conda/lib/python3.8/site-packages (from numba->nemo_toolkit[all]==1.5.1) (0.36.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from scikit-learn->nemo_toolkit[all]==1.5.1) (3.1.0)
Requirement already satisfied: defusedxml in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.7.1)
Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.5.13)
Requirement already satisfied: bleach in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (4.1.0)
Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.8.4)
Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (1.5.0)
Requirement already satisfied: testpath in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.6.0)
Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.8/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.1.2)
Requirement already satisfied: webencodings in /opt/conda/lib/python3.8/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.5.1)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas->nemo_toolkit[all]==1.5.1) (2021.3)
Requirement already satisfied: pip in /opt/conda/lib/python3.8/site-packages (from pip-api->isort[requirements]<5->nemo_toolkit[all]==1.5.1) (21.2.4)
Requirement already satisfied: yarg in /opt/conda/lib/python3.8/site-packages (from pipreqs->isort[requirements]<5->nemo_toolkit[all]==1.5.1) (0.1.9)
Requirement already satisfied: docopt in /opt/conda/lib/python3.8/site-packages (from pipreqs->isort[requirements]<5->nemo_toolkit[all]==1.5.1) (0.6.2)
Requirement already satisfied: simplejson>=3.8.1 in /opt/conda/lib/python3.8/site-packages (from pyannote.core->nemo_toolkit[all]==1.5.1) (3.17.6)
Requirement already satisfied: sortedcontainers>=2.0.4 in /opt/conda/lib/python3.8/site-packages (from pyannote.core->nemo_toolkit[all]==1.5.1) (2.4.0)
Requirement already satisfied: tabulate>=0.7.7 in /opt/conda/lib/python3.8/site-packages (from pyannote.metrics->nemo_toolkit[all]==1.5.1) (0.8.9)
Requirement already satisfied: pyannote.database>=4.0.1 in /opt/conda/lib/python3.8/site-packages (from pyannote.metrics->nemo_toolkit[all]==1.5.1) (4.1.3)
Requirement already satisfied: sympy>=1.1 in /opt/conda/lib/python3.8/site-packages (from pyannote.metrics->nemo_toolkit[all]==1.5.1) (1.10.1)
Requirement already satisfied: typer[all]>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from pyannote.database>=4.0.1->pyannote.metrics->nemo_toolkit[all]==1.5.1) (0.4.0)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy>=1.1->pyannote.metrics->nemo_toolkit[all]==1.5.1) (1.2.1)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in /opt/conda/lib/python3.8/site-packages (from typer[all]>=0.2.1->pyannote.database>=4.0.1->pyannote.metrics->nemo_toolkit[all]==1.5.1) (0.4.4)
Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in /opt/conda/lib/python3.8/site-packages (from typer[all]>=0.2.1->pyannote.database>=4.0.1->pyannote.metrics->nemo_toolkit[all]==1.5.1) (1.4.0)
Requirement already satisfied: py>=1.8.2 in /opt/conda/lib/python3.8/site-packages (from pytest->nemo_toolkit[all]==1.5.1) (1.11.0)
Requirement already satisfied: iniconfig in /opt/conda/lib/python3.8/site-packages (from pytest->nemo_toolkit[all]==1.5.1) (1.1.1)
Requirement already satisfied: pluggy<2.0,>=0.12 in /opt/conda/lib/python3.8/site-packages (from pytest->nemo_toolkit[all]==1.5.1) (1.0.0)
Requirement already satisfied: jarowinkler<1.1.0,>=1.0.2 in /opt/conda/lib/python3.8/site-packages (from rapidfuzz->nemo_toolkit[all]==1.5.1) (1.0.2)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /opt/conda/lib/python3.8/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.5.0->nemo_toolkit[all]==1.5.1) (1.7.1)
Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in /opt/conda/lib/python3.8/site-packages (from ruamel.yaml->nemo_toolkit[all]==1.5.1) (0.2.6)
Requirement already satisfied: portalocker in /opt/conda/lib/python3.8/site-packages (from sacrebleu[ja]->nemo_toolkit[all]==1.5.1) (2.4.0)
Requirement already satisfied: ipadic<2.0,>=1.0 in /opt/conda/lib/python3.8/site-packages (from sacrebleu[ja]->nemo_toolkit[all]==1.5.1) (1.0.0)
Requirement already satisfied: mecab-python3==1.0.3 in /opt/conda/lib/python3.8/site-packages (from sacrebleu[ja]->nemo_toolkit[all]==1.5.1) (1.0.3)
Requirement already satisfied: sphinxcontrib-htmlhelp>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (2.0.0)
Requirement already satisfied: alabaster<0.8,>=0.7 in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (0.7.12)
Requirement already satisfied: babel>=1.3 in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (2.9.1)
Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (1.1.5)
Requirement already satisfied: sphinxcontrib-devhelp in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (1.0.2)
Requirement already satisfied: sphinxcontrib-jsmath in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (1.0.1)
Requirement already satisfied: sphinxcontrib-qthelp in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (1.0.3)
Requirement already satisfied: snowballstemmer>=1.1 in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (2.2.0)
Requirement already satisfied: imagesize in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (1.3.0)
Requirement already satisfied: sphinxcontrib-applehelp in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (1.0.2)
Requirement already satisfied: docutils<0.18,>=0.14 in /opt/conda/lib/python3.8/site-packages (from sphinx->nemo_toolkit[all]==1.5.1) (0.17.1)
Requirement already satisfied: pybtex-docutils>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from sphinxcontrib-bibtex->nemo_toolkit[all]==1.5.1) (1.0.1)
Requirement already satisfied: pybtex>=0.24 in /opt/conda/lib/python3.8/site-packages (from sphinxcontrib-bibtex->nemo_toolkit[all]==1.5.1) (0.24.0)
Requirement already satisfied: latexcodec>=1.0.4 in /opt/conda/lib/python3.8/site-packages (from pybtex>=0.24->sphinxcontrib-bibtex->nemo_toolkit[all]==1.5.1) (2.0.1)
Requirement already satisfied: pure-eval in /opt/conda/lib/python3.8/site-packages (from stack-data->ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.2.2)
Requirement already satisfied: asttokens in /opt/conda/lib/python3.8/site-packages (from stack-data->ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (2.0.5)
Requirement already satisfied: executing in /opt/conda/lib/python3.8/site-packages (from stack-data->ipython>=4.0.0->ipywidgets->nemo_toolkit[all]==1.5.1) (0.8.3)
Requirement already satisfied: pathtools in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (0.1.2)
Requirement already satisfied: setproctitle in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (1.2.3)
Requirement already satisfied: GitPython>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (3.1.27)
Requirement already satisfied: sentry-sdk>=1.0.0 in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (1.5.10)
Requirement already satisfied: shortuuid>=0.5.0 in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (1.0.8)
Requirement already satisfied: docker-pycreds>=0.4.0 in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (0.4.0)
Requirement already satisfied: promise<3,>=2.0 in /opt/conda/lib/python3.8/site-packages (from wandb->nemo_toolkit[all]==1.5.1) (2.3)
Requirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.8/site-packages (from GitPython>=1.0.0->wandb->nemo_toolkit[all]==1.5.1) (4.0.9)
Requirement already satisfied: smmap<6,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb->nemo_toolkit[all]==1.5.1) (5.0.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
## 2. Download Citrinet model
Next, we download a pretrained Nemo Citrinet model and convert it to a Torchscript module:
[3]:
import nemo
import torch
import nemo.collections.asr as nemo_asr
from nemo.core import typecheck
typecheck.set_typecheck_enabled(False)
[4]:
variant = 'stt_en_citrinet_256'
print(f"Downloading and saving {variant}...")
asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name=variant)
asr_model.export(f"{variant}.ts")
Downloading and saving stt_en_citrinet_256...
[NeMo I 2022-04-21 23:12:45 cloud:56] Found existing object /root/.cache/torch/NeMo/NeMo_1.5.1/stt_en_citrinet_256/91a9cc5850784b2065e8a0aa3d526fd9/stt_en_citrinet_256.nemo.
[NeMo I 2022-04-21 23:12:45 cloud:62] Re-using file from: /root/.cache/torch/NeMo/NeMo_1.5.1/stt_en_citrinet_256/91a9cc5850784b2065e8a0aa3d526fd9/stt_en_citrinet_256.nemo
[NeMo I 2022-04-21 23:12:45 common:728] Instantiating model from pre-trained checkpoint
[NeMo I 2022-04-21 23:12:46 mixins:146] Tokenizer SentencePieceTokenizer initialized with 1024 tokens
[NeMo W 2022-04-21 23:12:47 modelPT:130] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
trim_silence: true
max_duration: 16.7
shuffle: true
is_tarred: false
tarred_audio_filepaths: null
use_start_end_token: false
[NeMo W 2022-04-21 23:12:47 modelPT:137] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
shuffle: false
use_start_end_token: false
[NeMo W 2022-04-21 23:12:47 modelPT:143] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
Test config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
shuffle: false
use_start_end_token: false
[NeMo I 2022-04-21 23:12:47 features:265] PADDING: 16
[NeMo I 2022-04-21 23:12:47 features:282] STFT using torch
[NeMo W 2022-04-21 23:12:47 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/features.py:315: FutureWarning: Pass sr=16000, n_fft=512 as keyword args. From version 0.10 passing these as positional arguments will result in an error
librosa.filters.mel(sample_rate, self.n_fft, n_mels=nfilt, fmin=lowfreq, fmax=highfreq), dtype=torch.float
[NeMo I 2022-04-21 23:12:49 save_restore_connector:149] Model EncDecCTCModelBPE was successfully restored from /root/.cache/torch/NeMo/NeMo_1.5.1/stt_en_citrinet_256/91a9cc5850784b2065e8a0aa3d526fd9/stt_en_citrinet_256.nemo.
[NeMo W 2022-04-21 23:12:49 export_utils:198] Swapped 0 modules
[NeMo W 2022-04-21 23:12:49 conv_asr:73] Turned off 235 masked convolutions
[NeMo W 2022-04-21 23:12:49 export_utils:198] Swapped 0 modules
[NeMo W 2022-04-21 23:12:50 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py:916: UserWarning: `optimize` is deprecated and has no effect. Use `with torch.jit.optimized_execution() instead
warnings.warn(
[NeMo W 2022-04-21 23:12:50 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torch/_jit_internal.py:668: LightningDeprecationWarning: The `LightningModule.model_size` property was deprecated in v1.5 and will be removed in v1.7. Please use the `pytorch_lightning.utilities.memory.get_model_size_mb`.
if hasattr(mod, name):
[NeMo W 2022-04-21 23:12:50 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torch/_jit_internal.py:669: LightningDeprecationWarning: The `LightningModule.model_size` property was deprecated in v1.5 and will be removed in v1.7. Please use the `pytorch_lightning.utilities.memory.get_model_size_mb`.
item = getattr(mod, name)
[NeMo W 2022-04-21 23:12:50 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torch/_jit_internal.py:668: LightningDeprecationWarning: `LightningModule.use_amp` was deprecated in v1.6 and will be removed in v1.8. Please use `Trainer.amp_backend`.
if hasattr(mod, name):
[NeMo W 2022-04-21 23:12:50 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torch/_jit_internal.py:669: LightningDeprecationWarning: `LightningModule.use_amp` was deprecated in v1.6 and will be removed in v1.8. Please use `Trainer.amp_backend`.
item = getattr(mod, name)
[4]:
(['stt_en_citrinet_256.ts'],
['nemo.collections.asr.models.ctc_bpe_models.EncDecCTCModelBPE exported to ONNX'])
Benchmark utility ¶
Let us define a helper benchmarking function, then benchmark the original Pytorch model.
[5]:
from __future__ import print_function
from __future__ import absolute_import
from __future__ import division
import argparse
import timeit
import numpy as np
import torch
import torch_tensorrt as trtorch
import torch.backends.cudnn as cudnn
def benchmark(model, input_tensor, num_loops, model_name, batch_size):
def timeGraph(model, input_tensor, num_loops):
print("Warm up ...")
with torch.no_grad():
for _ in range(20):
features = model(input_tensor)
torch.cuda.synchronize()
print("Start timing ...")
timings = []
with torch.no_grad():
for i in range(num_loops):
start_time = timeit.default_timer()
features = model(input_tensor)
torch.cuda.synchronize()
end_time = timeit.default_timer()
timings.append(end_time - start_time)
# print("Iteration {}: {:.6f} s".format(i, end_time - start_time))
return timings
def printStats(graphName, timings, batch_size):
times = np.array(timings)
steps = len(times)
speeds = batch_size / times
time_mean = np.mean(times)
time_med = np.median(times)
time_99th = np.percentile(times, 99)
time_std = np.std(times, ddof=0)
speed_mean = np.mean(speeds)
speed_med = np.median(speeds)
msg = ("\n%s =================================\n"
"batch size=%d, num iterations=%d\n"
" Median samples/s: %.1f, mean: %.1f\n"
" Median latency (s): %.6f, mean: %.6f, 99th_p: %.6f, std_dev: %.6f\n"
) % (graphName,
batch_size, steps,
speed_med, speed_mean,
time_med, time_mean, time_99th, time_std)
print(msg)
timings = timeGraph(model, input_tensor, num_loops)
printStats(model_name, timings, batch_size)
precisions_str = 'fp32' # Precision (default=fp32, fp16)
variant = 'stt_en_citrinet_256' # Nemo Citrinet variant
batch_sizes = [1, 8, 32, 128] # Batch sizes (default=1,8,32,128)
trt = False # If True, infer with Torch-TensorRT engine. Else, infer with Pytorch model.
precision = torch.float32 if precisions_str =='fp32' else torch.float16
for batch_size in batch_sizes:
if trt:
model_name = f"{variant}_bs{batch_size}_{precision}.torch-tensorrt"
else:
model_name = f"{variant}.ts"
print(f"Loading model: {model_name}")
# Load traced model to CPU first
model = torch.jit.load(model_name).cuda()
cudnn.benchmark = True
# Create random input tensor of certain size
torch.manual_seed(12345)
input_shape=(batch_size, 80, 1488)
input_tensor = torch.randn(input_shape).cuda()
# Timing graph inference
benchmark(model, input_tensor, 50, model_name, batch_size)
Loading model: stt_en_citrinet_256.ts
Warm up ...
Start timing ...
stt_en_citrinet_256.ts =================================
batch size=1, num iterations=50
Median samples/s: 102.0, mean: 102.0
Median latency (s): 0.009802, mean: 0.009803, 99th_p: 0.009836, std_dev: 0.000014
Loading model: stt_en_citrinet_256.ts
Warm up ...
Start timing ...
stt_en_citrinet_256.ts =================================
batch size=8, num iterations=50
Median samples/s: 429.1, mean: 429.1
Median latency (s): 0.018642, mean: 0.018643, 99th_p: 0.018670, std_dev: 0.000014
Loading model: stt_en_citrinet_256.ts
Warm up ...
Start timing ...
stt_en_citrinet_256.ts =================================
batch size=32, num iterations=50
Median samples/s: 551.3, mean: 551.2
Median latency (s): 0.058047, mean: 0.058053, 99th_p: 0.058375, std_dev: 0.000106
Loading model: stt_en_citrinet_256.ts
Warm up ...
Start timing ...
stt_en_citrinet_256.ts =================================
batch size=128, num iterations=50
Median samples/s: 594.1, mean: 594.1
Median latency (s): 0.215434, mean: 0.215446, 99th_p: 0.215806, std_dev: 0.000116
Confirming the GPU we are using here:
[6]:
!nvidia-smi
Thu Apr 21 23:13:32 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA TITAN V On | 00000000:17:00.0 Off | N/A |
| 38% 55C P2 42W / 250W | 2462MiB / 12288MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA TITAN V On | 00000000:65:00.0 Off | N/A |
| 28% 39C P8 26W / 250W | 112MiB / 12288MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3909 G 4MiB |
| 0 N/A N/A 6047 C 2453MiB |
| 1 N/A N/A 3909 G 39MiB |
| 1 N/A N/A 4161 G 67MiB |
+-----------------------------------------------------------------------------+
## 3. Create Torch-TensorRT modules
In this step, we optimize the Citrinet Torchscript module with Torch-TensorRT with various precisions and batch sizes.
[10]:
import torch
import torch.nn as nn
import torch_tensorrt as torchtrt
import argparse
variant = "stt_en_citrinet_256"
precisions = [torch.float, torch.half]
batch_sizes = [1,8,32,128]
model = torch.jit.load(f"{variant}.ts")
for precision in precisions:
for batch_size in batch_sizes:
compile_settings = {
"inputs": [torchtrt.Input(shape=[batch_size, 80, 1488])],
"enabled_precisions": {precision},
"workspace_size": 2000000000,
"truncate_long_and_double": True,
}
print(f"Generating Torchscript-TensorRT module for batchsize {batch_size} precision {precision}")
trt_ts_module = torchtrt.compile(model, **compile_settings)
torch.jit.save(trt_ts_module, f"{variant}_bs{batch_size}_{precision}.torch-tensorrt")
Generating Torchscript-TensorRT module for batchsize 1 precision torch.float32
Generating Torchscript-TensorRT module for batchsize 8 precision torch.float32
Generating Torchscript-TensorRT module for batchsize 32 precision torch.float32
Generating Torchscript-TensorRT module for batchsize 128 precision torch.float32
Generating Torchscript-TensorRT module for batchsize 1 precision torch.float16
Generating Torchscript-TensorRT module for batchsize 8 precision torch.float16
Generating Torchscript-TensorRT module for batchsize 32 precision torch.float16
Generating Torchscript-TensorRT module for batchsize 128 precision torch.float16
## 4. Benchmark Torch-TensorRT models
Finally, we are ready to benchmark the Torch-TensorRT optimized Citrinet models.
FP32 (single precision) ¶
[13]:
precisions_str = 'fp32' # Precision (default=fp32, fp16)
batch_sizes = [1, 8, 32, 128] # Batch sizes (default=1,8,32,128)
precision = torch.float32 if precisions_str =='fp32' else torch.float16
trt = True
for batch_size in batch_sizes:
if trt:
model_name = f"{variant}_bs{batch_size}_{precision}.torch-tensorrt"
else:
model_name = f"{variant}.ts"
print(f"Loading model: {model_name}")
# Load traced model to CPU first
model = torch.jit.load(model_name).cuda()
cudnn.benchmark = True
# Create random input tensor of certain size
torch.manual_seed(12345)
input_shape=(batch_size, 80, 1488)
input_tensor = torch.randn(input_shape).cuda()
# Timing graph inference
benchmark(model, input_tensor, 50, model_name, batch_size)
Loading model: stt_en_citrinet_256_bs1_torch.float32.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs1_torch.float32.torch-tensorrt =================================
batch size=1, num iterations=50
Median samples/s: 242.2, mean: 218.0
Median latency (s): 0.004128, mean: 0.004825, 99th_p: 0.008071, std_dev: 0.001270
Loading model: stt_en_citrinet_256_bs8_torch.float32.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs8_torch.float32.torch-tensorrt =================================
batch size=8, num iterations=50
Median samples/s: 729.9, mean: 709.0
Median latency (s): 0.010961, mean: 0.011388, 99th_p: 0.016114, std_dev: 0.001256
Loading model: stt_en_citrinet_256_bs32_torch.float32.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs32_torch.float32.torch-tensorrt =================================
batch size=32, num iterations=50
Median samples/s: 955.6, mean: 953.4
Median latency (s): 0.033488, mean: 0.033572, 99th_p: 0.035722, std_dev: 0.000545
Loading model: stt_en_citrinet_256_bs128_torch.float32.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs128_torch.float32.torch-tensorrt =================================
batch size=128, num iterations=50
Median samples/s: 1065.8, mean: 1069.4
Median latency (s): 0.120097, mean: 0.119708, 99th_p: 0.121618, std_dev: 0.001260
FP16 (half precision) ¶
[14]:
precisions_str = 'fp16' # Precision (default=fp32, fp16)
batch_sizes = [1, 8, 32, 128] # Batch sizes (default=1,8,32,128)
precision = torch.float32 if precisions_str =='fp32' else torch.float16
for batch_size in batch_sizes:
if trt:
model_name = f"{variant}_bs{batch_size}_{precision}.torch-tensorrt"
else:
model_name = f"{variant}.ts"
print(f"Loading model: {model_name}")
# Load traced model to CPU first
model = torch.jit.load(model_name).cuda()
cudnn.benchmark = True
# Create random input tensor of certain size
torch.manual_seed(12345)
input_shape=(batch_size, 80, 1488)
input_tensor = torch.randn(input_shape).cuda()
# Timing graph inference
benchmark(model, input_tensor, 50, model_name, batch_size)
Loading model: stt_en_citrinet_256_bs1_torch.float16.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs1_torch.float16.torch-tensorrt =================================
batch size=1, num iterations=50
Median samples/s: 288.9, mean: 272.9
Median latency (s): 0.003462, mean: 0.003774, 99th_p: 0.006846, std_dev: 0.000820
Loading model: stt_en_citrinet_256_bs8_torch.float16.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs8_torch.float16.torch-tensorrt =================================
batch size=8, num iterations=50
Median samples/s: 1201.0, mean: 1190.9
Median latency (s): 0.006661, mean: 0.006733, 99th_p: 0.008453, std_dev: 0.000368
Loading model: stt_en_citrinet_256_bs32_torch.float16.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs32_torch.float16.torch-tensorrt =================================
batch size=32, num iterations=50
Median samples/s: 1538.2, mean: 1516.4
Median latency (s): 0.020804, mean: 0.021143, 99th_p: 0.024492, std_dev: 0.000973
Loading model: stt_en_citrinet_256_bs128_torch.float16.torch-tensorrt
Warm up ...
Start timing ...
stt_en_citrinet_256_bs128_torch.float16.torch-tensorrt =================================
batch size=128, num iterations=50
Median samples/s: 1792.0, mean: 1777.0
Median latency (s): 0.071428, mean: 0.072057, 99th_p: 0.076796, std_dev: 0.001351
## 5. Conclusion
In this notebook, we have walked through the complete process of optimizing the Citrinet model with Torch-TensorRT. On an A100 GPU, with Torch-TensorRT, we observe a speedup of ~ 2.4X with FP32, and ~ 2.9X with FP16 at batchsize of 128.
What’s next ¶
Now it’s time to try Torch-TensorRT on your own model. Fill out issues at https://github.com/NVIDIA/Torch-TensorRT . Your involvement will help future development of Torch-TensorRT.
[ ]: