This document is the API specification for the Deep Java Library (DJL) Huggingface tokenizers extension.
The Tokeniszers module contains the DJL extension for Huggingface tokenizers.