Models

<tokenizerslangcontent> <python> ## BPE

[[autodoc]] tokenizers.models.BPE

Model

[[autodoc]] tokenizers.models.Model

Unigram

[[autodoc]] tokenizers.models.Unigram

WordLevel

[[autodoc]] tokenizers.models.WordLevel

WordPiece

[[autodoc]] tokenizers.models.WordPiece </python> <rust> The Rust API Reference is available directly on the Docs.rs website. </rust> <node> The node API has not been documented yet. </node> </tokenizerslangcontent>