Back to Tokenizers

Decoders

docs/source-doc-builder/api/decoders.mdx

0.23.1873 B
Original Source

Decoders

<tokenizerslangcontent> <python> ## DecodeStream [[autodoc]] tokenizers.decoders.DecodeStream

BPEDecoder

[[autodoc]] tokenizers.decoders.BPEDecoder

ByteFallback

[[autodoc]] tokenizers.decoders.ByteFallback

ByteLevel

[[autodoc]] tokenizers.decoders.ByteLevel

CTC

[[autodoc]] tokenizers.decoders.CTC

Fuse

[[autodoc]] tokenizers.decoders.Fuse

Metaspace

[[autodoc]] tokenizers.decoders.Metaspace

Replace

[[autodoc]] tokenizers.decoders.Replace

Sequence

[[autodoc]] tokenizers.decoders.Sequence

Strip

[[autodoc]] tokenizers.decoders.Strip

WordPiece

[[autodoc]] tokenizers.decoders.WordPiece

</python> <rust> The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. </rust> <node> The node API has not been documented yet. </node> </tokenizerslangcontent>