What is a tokenizer?

A tokenizer is a class that splits a string into tokens and encodes them into numerical(int) values.

The Tokenizer Contract

We can simply use the tokenizer from Microsoft.ML.Tokenizer package