13fromtokenizersimportTokenizer1415fromlabmlimportlab,monit

[email protected]('Load NeoX Tokenizer')19defget\_tokenizer()-\>Tokenizer:

25vocab\_file=lab.get\_data\_path()/'neox'/'slim\_weights'/'20B\_tokenizer.json'26tokenizer=Tokenizer.from\_file(str(vocab\_file))2728returntokenizer