Back to Paradedb

Whitespace

docs/documentation/tokenizers/available-tokenizers/whitespace.mdx

0.23.3493 B
Original Source

The whitespace tokenizer splits only on whitespace. It also lowercases characters by default.

sql
CREATE INDEX search_idx ON mock_items
USING bm25 (id, (description::pdb.whitespace))
WITH (key_field='id');

To get a feel for this tokenizer, run the following command and replace the text with your own:

sql
SELECT 'Tokenize me!'::pdb.whitespace::text[];
ini
      text
----------------
 {tokenize,me!}
(1 row)