Quicktour

Let's have a quick look at the 🤗 Tokenizers library features. The library provides an implementation of today's most used tokenizers that is both easy to use and blazing fast.

.. only:: python

It can be used to instantiate a :ref:`pretrained tokenizer <pretrained>` but we will start our
quicktour by building one from scratch and see how we can train it.

Build a tokenizer from scratch

To illustrate how fast the 🤗 Tokenizers library is, let's train a new tokenizer on wikitext-103 <https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/>__ (516M of text) in just a few seconds. First things first, you will need to download this dataset and unzip it with:

.. code-block:: bash

wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip
unzip wikitext-103-raw-v1.zip

Training the tokenizer


.. entities:: python

    BpeTrainer
        :class:`~tokenizers.trainers.BpeTrainer`
    vocab_size
        :obj:`vocab_size`
    min_frequency
        :obj:`min_frequency`
    special_tokens
        :obj:`special_tokens`
    unk_token
        :obj:`unk_token`
    pad_token
        :obj:`pad_token`

.. entities:: rust

    BpeTrainer
        :rust_struct:`~tokenizers::models::bpe::BpeTrainer`
    vocab_size
        :obj:`vocab_size`
    min_frequency
        :obj:`min_frequency`
    special_tokens
        :obj:`special_tokens`
    unk_token
        :obj:`unk_token`
    pad_token
        :obj:`pad_token`

.. entities:: node

    BpeTrainer
        BpeTrainer
    vocab_size
        :obj:`vocabSize`
    min_frequency
        :obj:`minFrequency`
    special_tokens
        :obj:`specialTokens`
    unk_token
        :obj:`unkToken`
    pad_token
        :obj:`padToken`

In this tour, we will build and train a Byte-Pair Encoding (BPE) tokenizer. For more information
about the different type of tokenizers, check out this `guide
<https://huggingface.co/docs/transformers/main/en/tokenizer_summary#summary-of-the-tokenizers>`__ in the 🤗 Transformers
documentation. Here, training the tokenizer means it will learn merge rules by:

- Start with all the characters present in the training corpus as tokens.
- Identify the most common pair of tokens and merge it into one token.
- Repeat until the vocabulary (e.g., the number of tokens) has reached the size we want.

The main API of the library is the :entity:`class` :entity:`Tokenizer`, here is how we instantiate
one with a BPE model:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START init_tokenizer
        :end-before: END init_tokenizer
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_init_tokenizer
        :end-before: END quicktour_init_tokenizer
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START init_tokenizer
        :end-before: END init_tokenizer
        :dedent: 4

To train our tokenizer on the wikitext files, we will need to instantiate a `trainer`, in this case
a :entity:`BpeTrainer`

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START init_trainer
        :end-before: END init_trainer
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_init_trainer
        :end-before: END quicktour_init_trainer
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START init_trainer
        :end-before: END init_trainer
        :dedent: 4

We can set the training arguments like :entity:`vocab_size` or :entity:`min_frequency` (here left at
their default values of 30,000 and 0) but the most important part is to give the
:entity:`special_tokens` we plan to use later on (they are not used at all during training) so that
they get inserted in the vocabulary.

.. note::

    The order in which you write the special tokens list matters: here :obj:`"[UNK]"` will get the
    ID 0, :obj:`"[CLS]"` will get the ID 1 and so forth.

We could train our tokenizer right now, but it wouldn't be optimal. Without a pre-tokenizer that
will split our inputs into words, we might get tokens that overlap several words: for instance we
could get an :obj:`"it is"` token since those two words often appear next to each other. Using a
pre-tokenizer will ensure no token is bigger than a word returned by the pre-tokenizer. Here we want
to train a subword BPE tokenizer, and we will use the easiest pre-tokenizer possible by splitting
on whitespace.

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START init_pretok
        :end-before: END init_pretok
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_init_pretok
        :end-before: END quicktour_init_pretok
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START init_pretok
        :end-before: END init_pretok
        :dedent: 4

Now, we can just call the :entity:`Tokenizer.train` method with any list of files we want
to use:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START train
        :end-before: END train
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_train
        :end-before: END quicktour_train
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START train
        :end-before: END train
        :dedent: 4

This should only take a few seconds to train our tokenizer on the full wikitext dataset!
To save the tokenizer in one file that contains all its configuration and vocabulary, just use the
:entity:`Tokenizer.save` method:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START save
        :end-before: END save
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_save
        :end-before: END quicktour_save
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START save
        :end-before: END save
        :dedent: 4

and you can reload your tokenizer from that file with the :entity:`Tokenizer.from_file`
:entity:`classmethod`:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START reload_tokenizer
        :end-before: END reload_tokenizer
        :dedent: 12

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_reload_tokenizer
        :end-before: END quicktour_reload_tokenizer
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START reload_tokenizer
        :end-before: END reload_tokenizer
        :dedent: 4

Using the tokenizer

Now that we have trained a tokenizer, we can use it on any text we want with the :entity:Tokenizer.encode method:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START encode
    :end-before: END encode
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_encode
    :end-before: END quicktour_encode
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START encode
    :end-before: END encode
    :dedent: 4

This applied the full pipeline of the tokenizer on the text, returning an :entity:Encoding object. To learn more about this pipeline, and how to apply (or customize) parts of it, check out :doc:this page <pipeline>.

This :entity:Encoding object then has all the attributes you need for your deep learning model (or other). The :obj:tokens attribute contains the segmentation of your text in tokens:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START print_tokens
    :end-before: END print_tokens
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_print_tokens
    :end-before: END quicktour_print_tokens
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START print_tokens
    :end-before: END print_tokens
    :dedent: 4

Similarly, the :obj:ids attribute will contain the index of each of those tokens in the tokenizer's vocabulary:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START print_ids
    :end-before: END print_ids
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_print_ids
    :end-before: END quicktour_print_ids
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START print_ids
    :end-before: END print_ids
    :dedent: 4

An important feature of the 🤗 Tokenizers library is that it comes with full alignment tracking, meaning you can always get the part of your original sentence that corresponds to a given token. Those are stored in the :obj:offsets attribute of our :entity:Encoding object. For instance, let's assume we would want to find back what caused the :obj:"[UNK]" token to appear, which is the token at index 9 in the list, we can just ask for the offset at the index:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START print_offsets
    :end-before: END print_offsets
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_print_offsets
    :end-before: END quicktour_print_offsets
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START print_offsets
    :end-before: END print_offsets
    :dedent: 4

and those are the indices that correspond to the emoji in the original sentence:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START use_offsets
    :end-before: END use_offsets
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_use_offsets
    :end-before: END quicktour_use_offsets
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START use_offsets
    :end-before: END use_offsets
    :dedent: 4

Post-processing


We might want our tokenizer to automatically add special tokens, like :obj:`"[CLS]"` or
:obj:`"[SEP]"`. To do this, we use a post-processor. :entity:`TemplateProcessing` is the
most commonly used, you just have to specify a template for the processing of single sentences and
pairs of sentences, along with the special tokens and their IDs.

When we built our tokenizer, we set :obj:`"[CLS]"` and :obj:`"[SEP]"` in positions 1 and 2 of our
list of special tokens, so this should be their IDs. To double-check, we can use the
:entity:`Tokenizer.token_to_id` method:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START check_sep
        :end-before: END check_sep
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_check_sep
        :end-before: END quicktour_check_sep
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START check_sep
        :end-before: END check_sep
        :dedent: 4

Here is how we can set the post-processing to give us the traditional BERT inputs:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START init_template_processing
        :end-before: END init_template_processing
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_init_template_processing
        :end-before: END quicktour_init_template_processing
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START init_template_processing
        :end-before: END init_template_processing
        :dedent: 4

Let's go over this snippet of code in more details. First we specify the template for single
sentences: those should have the form :obj:`"[CLS] $A [SEP]"` where :obj:`$A` represents our
sentence.

Then, we specify the template for sentence pairs, which should have the form
:obj:`"[CLS] $A [SEP] $B [SEP]"` where :obj:`$A` represents the first sentence and :obj:`$B` the
second one. The :obj:`:1` added in the template represent the `type IDs` we want for each part of
our input: it defaults to 0 for everything (which is why we don't have :obj:`$A:0`) and here we set
it to 1 for the tokens of the second sentence and the last :obj:`"[SEP]"` token.

Lastly, we specify the special tokens we used and their IDs in our tokenizer's vocabulary.

To check out this worked properly, let's try to encode the same sentence as before:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START print_special_tokens
        :end-before: END print_special_tokens
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_print_special_tokens
        :end-before: END quicktour_print_special_tokens
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START print_special_tokens
        :end-before: END print_special_tokens
        :dedent: 4

To check the results on a pair of sentences, we just pass the two sentences to
:entity:`Tokenizer.encode`:

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START print_special_tokens_pair
        :end-before: END print_special_tokens_pair
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_print_special_tokens_pair
        :end-before: END quicktour_print_special_tokens_pair
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START print_special_tokens_pair
        :end-before: END print_special_tokens_pair
        :dedent: 4

You can then check the type IDs attributed to each token is correct with

.. only:: python

    .. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
        :language: python
        :start-after: START print_type_ids
        :end-before: END print_type_ids
        :dedent: 8

.. only:: rust

    .. literalinclude:: ../../tokenizers/tests/documentation.rs
        :language: rust
        :start-after: START quicktour_print_type_ids
        :end-before: END quicktour_print_type_ids
        :dedent: 4

.. only:: node

    .. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
        :language: javascript
        :start-after: START print_type_ids
        :end-before: END print_type_ids
        :dedent: 4

If you save your tokenizer with :entity:`Tokenizer.save`, the post-processor will be saved along.

Encoding multiple sentences in a batch

To get the full speed of the 🤗 Tokenizers library, it's best to process your texts by batches by using the :entity:Tokenizer.encode_batch method:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START encode_batch
    :end-before: END encode_batch
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_encode_batch
    :end-before: END quicktour_encode_batch
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START encode_batch
    :end-before: END encode_batch
    :dedent: 4

The output is then a list of :entity:Encoding objects like the ones we saw before. You can process together as many texts as you like, as long as it fits in memory.

To process a batch of sentences pairs, pass two lists to the :entity:Tokenizer.encode_batch method: the list of sentences A and the list of sentences B:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START encode_batch_pair
    :end-before: END encode_batch_pair
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_encode_batch_pair
    :end-before: END quicktour_encode_batch_pair
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START encode_batch_pair
    :end-before: END encode_batch_pair
    :dedent: 4

When encoding multiple sentences, you can automatically pad the outputs to the longest sentence present by using :entity:Tokenizer.enable_padding, with the :entity:pad_token and its ID (which we can double-check the id for the padding token with :entity:Tokenizer.token_to_id like before):

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START enable_padding
    :end-before: END enable_padding
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_enable_padding
    :end-before: END quicktour_enable_padding
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START enable_padding
    :end-before: END enable_padding
    :dedent: 4

We can set the :obj:direction of the padding (defaults to the right) or a given :obj:length if we want to pad every sample to that specific number (here we leave it unset to pad to the size of the longest text).

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START print_batch_tokens
    :end-before: END print_batch_tokens
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_print_batch_tokens
    :end-before: END quicktour_print_batch_tokens
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START print_batch_tokens
    :end-before: END print_batch_tokens
    :dedent: 4

In this case, the attention mask generated by the tokenizer takes the padding into account:

.. only:: python

.. literalinclude:: ../../bindings/python/tests/documentation/test_quicktour.py
    :language: python
    :start-after: START print_attention_mask
    :end-before: END print_attention_mask
    :dedent: 8

.. only:: rust

.. literalinclude:: ../../tokenizers/tests/documentation.rs
    :language: rust
    :start-after: START quicktour_print_attention_mask
    :end-before: END quicktour_print_attention_mask
    :dedent: 4

.. only:: node

.. literalinclude:: ../../bindings/node/examples/documentation/quicktour.test.ts
    :language: javascript
    :start-after: START print_attention_mask
    :end-before: END print_attention_mask
    :dedent: 4

.. _pretrained:

.. only:: python

Using a pretrained tokenizer
------------------------------------------------------------------------------------------------

You can load any tokenizer from the Hugging Face Hub as long as a `tokenizer.json` file is
available in the repository.

.. code-block:: python

    from tokenizers import Tokenizer

    tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

Importing a pretrained tokenizer from legacy vocabulary files
------------------------------------------------------------------------------------------------

You can also import a pretrained tokenizer directly in, as long as you have its vocabulary file.
For instance, here is how to import the classic pretrained BERT tokenizer:

.. code-block:: python

    from tokenizers import BertWordPieceTokenizer

    tokenizer = BertWordPieceTokenizer("bert-base-uncased-vocab.txt", lowercase=True)

as long as you have downloaded the file `bert-base-uncased-vocab.txt` with

.. code-block:: bash

    wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt