Normalizers

<tokenizerslangcontent> <python>

ByteLevel

[[autodoc]] tokenizers.normalizers.ByteLevel

Lowercase

[[autodoc]] tokenizers.normalizers.Lowercase

NFC

[[autodoc]] tokenizers.normalizers.NFC

NFD

[[autodoc]] tokenizers.normalizers.NFD

NFKC

[[autodoc]] tokenizers.normalizers.NFKC

NFKD

[[autodoc]] tokenizers.normalizers.NFKD

Nmt

[[autodoc]] tokenizers.normalizers.Nmt

Normalizer

[[autodoc]] tokenizers.normalizers.Normalizer

Precompiled

[[autodoc]] tokenizers.normalizers.Precompiled

Replace

[[autodoc]] tokenizers.normalizers.Replace

Sequence

[[autodoc]] tokenizers.normalizers.Sequence

Strip

[[autodoc]] tokenizers.normalizers.Strip

StripAccents

[[autodoc]] tokenizers.normalizers.StripAccents

BertNormalizer

[[autodoc]] tokenizers.normalizers.BertNormalizer

</python> <rust> The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. </rust> <node> The node API has not been documented yet. </node> </tokenizerslangcontent>