Introduction

• ./convert_tokenizer.py It produces ./token_scores.json and ./vocab.json from ./tokenizer.model

• ./test_tokenizer.py is used to test the exported ./token_scores.json and ./vocab.json

In C++, we don't need to use the sentencepiece or protobuf for the tokenizer.