Class: Tokenizer

Tokenizer(settings)

Base class for a tokenizer

Constructor

new Tokenizer(settings)

Constructor of the class.
Parameters:
Name Type Description
settings Object Settings for this instance.
Source:

Methods

trim(arr) → {Array.<string>}

Remove empty items at left at right of an array.
Parameters:
Name Type Description
arr Array.<string> Array to be trimmed.
Source:
Returns:
Trimmed array.
Type
Array.<string>