Tokenizer

<tokenizerslangcontent> <python> ## Tokenizer

[[autodoc]] tokenizers.Tokenizer - all - decoder - model - normalizer - padding - post_processor - pre_tokenizer - truncation </python> <rust> The Rust API Reference is available directly on the Docs.rs website. </rust> <node> The node API has not been documented yet. </node> </tokenizerslangcontent>