import SystemTableCloud from '@site/docs/_snippets/_system_table_cloud.md';

system.tokenizers

Shows all available tokenizers. These can be used in functions tokens, hasAllTokens, hasAnyTokens, and the text index.

Columns:

Example

sql

SELECT * FROM system.tokenizers;

text

┌─name────────────┐
│ ngrams          │
│ splitByNonAlpha │
│ sparseGrams     │
│ tokenbf_v1      │
│ ngrambf_v1      │
│ array           │
│ splitByString   │
│ sparse_grams    │
│ unicode_word    │
└─────────────────┘