ContextQMD
Back to Sonic

Sonic Configuration

CONFIGURATION.md

1.7.49.6 KB
Original Source

Sonic Configuration

Configuration sources

Sonic looks for its configuration as TOML in ./config.cfg, or whatever you passed via --config/-c.

This file is optional, as all configuration keys can be defined using SONIC_ environment variables (which take precedence over the static configuration). The path separator is __, which means foo.bar.baz will be read from SONIC_FOO__BAR__BAZ.

Configuration keys reference

All available configuration options are commented below, with allowed values:

Server configuration

Under [server]:

  • log_level (type: string, allowed: debug, info, warn, error, default: error) — Verbosity of logging, set it to error in production

Channel configuration

Under [channel]:

  • inet (type: string, allowed: IPv4 / IPv6 + port, default: [::1]:1491) — Host and TCP port Sonic Channel should listen on
  • tcp_timeout (type: integer, allowed: seconds, default: 300) — Timeout of idle/dead client connections to Sonic Channel
  • auth_password (type: string, allowed: password values, default: none) — Authentication password required to connect to the channel (optional but recommended)

channel.search has been deprecated in favor of search, but it’s kept as an alias for backward compatibility reasons.

Normalization configuration

Warning: After making changes to normalization steps, you must rebuild Sonic’s index by re-ingesting all data. To avoid such breaking change, new features are disabled by default (opt-in). On major releases, some steps might become enabled by default (opt-out). Sonic won’t enable unreasonable defaults, but override if you need consistency.

Under [normalization]:

  • diacritic_folding_enabled (type: boolean, allowed: true, false, default: false) — Whether to enable diacritic folding or not (it reduces the index size and improves results)
  • stemming_enabled (type: boolean, allowed: true, false, default: false) — Whether to enable stemming or not (it avoids losing non-flushed data in case of server crash)
    • Warning: Enabling stemming greatly affects the quality of Sonic results. Enable only if you have a good reason to.

Tokenization configuration

Warning: After making changes to tokenization steps, you must rebuild Sonic’s index by re-ingesting all data. To avoid such breaking change, new features are disabled by default (opt-in). On major releases, some steps might become enabled by default (opt-out). Sonic won’t enable unreasonable defaults, but override if you need consistency.

Under [tokenization]:

  • detect_special_patterns (type: boolean, allowed: true, false, default: true) — Whether the tokenizer should detect special patterns or not
    • Sonic does fuzzy matching by default. However, some search terms are usually expected to match exactly, like email addresses. To support this use case, Sonic detects common patterns (e.g. email addresses, phone numbers, UUIDs, etc. and ensures they are both not split by the tokenizer (unless tokenization.compat_split_special_patterns = true) and matched exactly in queries.
    • For more information, see docs/tokenizer-pattern-matching.md.
    • This feature adds negligible overhead, you should probably not disable it.
  • compat_split_special_patterns (type: boolean, allowed: true, false, default: true) — Whether the tokenizer should split special patterns or not
    • Special patterns are matched exactly when performing a query. However, doing so without rebuilding your Sonic index breaks queries with special patterns. This flag enables a compatibility feature that integrates with an existing inex (at the cost of potentially worse results).
    • For more information, see docs/tokenizer-pattern-matching.md.
    • You don’t need to rebuild your Sonic index if you use tokenization.compat_split_special_patterns = true (default).
    • If you can easily rebuild your Sonic index and sometimes query things like email addresses, phone numbers or identifiers, it is recommended that you disable this feature.

Search configuration

Under [search]:

  • query_limit_default (type: integer, allowed: numbers, default: 10) — Default search results limit for a query command (if the LIMIT command modifier is not used when issuing a QUERY command)
  • query_limit_maximum (type: integer, allowed: numbers, default: 100) — Maximum search results limit for a query command (if the LIMIT command modifier is being used when issuing a QUERY command)
  • query_alternates_try (type: integer, allowed: numbers, default: 4) — Number of alternate words that look like query word to try if there are not enough query results (if zero, no alternate will be tried; if too high there may be a noticeable performance penalty)
  • suggest_limit_default (type: integer, allowed: numbers, default: 5) — Default suggested words limit for a suggest command (if the LIMIT command modifier is not used when issuing a SUGGEST command)
  • suggest_limit_maximum (type: integer, allowed: numbers, default: 20) — Maximum suggested words limit for a suggest command (if the LIMIT command modifier is being used when issuing a SUGGEST command)
  • list_limit_default (type: integer, allowed: numbers, default: 100) — Default listed words limit for a list command (if the LIMIT command modifier is not used when issuing a LIST command)
  • list_limit_maximum (type: integer, allowed: numbers, default: 500) — Maximum listed words limit for a list command (if the LIMIT command modifier is being used when issuing a LIST command)

KV store configuration

Under [store.kv]:

  • path (type: string, allowed: UNIX path, default: ./data/store/kv/) — Path to the Key-Value database store

  • retain_word_objects (type: integer, allowed: numbers, default: 1000) — Maximum number of objects a given word in the index can be linked to (older objects are cleared using a sliding window)

  • pool.inactive_after (type: integer, allowed: seconds, default: 1800) — Time after which a cached database is considered inactive and can be closed (if it is not used, ie. re-activated)

  • database.flush_after (type: integer, allowed: seconds, default: 900) — Time after which pending database updates should be flushed from memory to disk (increase this delay if you encounter high-CPU usage issues when a flush task kicks-in; this value should be lower than store.kv.pool.inactive_after)

  • database.compress (type: boolean, allowed: true, false, default: true) — Whether to compress database or not (uses Zstandard)

  • database.parallelism (type: integer, allowed: numbers, default: 2) — Limit on the number of compaction and flush threads that can run at the same time

  • database.max_files (type: integer, allowed: numbers, no default) — Maximum number of database files kept open at the same time per-database (if any; otherwise there are no limits)

  • database.max_compactions (type: integer, allowed: numbers, default: 1) — Limit on the number of concurrent database compaction jobs

  • database.max_flushes (type: integer, allowed: numbers, default: 1) — Limit on the number of concurrent database flush jobs

  • database.write_buffer (type: integer, allowed: numbers, default: 16384) — Maximum size in KB of the database write buffer, after which data gets flushed to disk (ie. 16384 is 16MB; the size should be a multiple of 1024, eg. 128 * 1024 = 131072 for 128MB)

  • database.write_ahead_log (type: boolean, allowed: true, false, default: true) — Whether to enable Write-Ahead Log or not (it avoids losing non-flushed data in case of server crash)

FST store configuration

Under [store.fst]:

  • path (type: string, allowed: UNIX path, default: ./data/store/fst/) — Path to the Finite-State Transducer database store

  • pool.inactive_after (type: integer, allowed: seconds, default: 300) — Time after which a cached graph is considered inactive and can be closed (if it is not used, ie. re-activated)

  • graph.consolidate_after (type: integer, allowed: seconds, default: 180) — Time after which a graph that has pending updates should be consolidated (increase this delay if you encounter high-CPU usage issues when a consolidation task kicks-in; this value should be lower than store.fst.pool.inactive_after)

  • graph.max_size (type: integer, allowed: numbers, default: 2048) — Maximum size in KB of the graph file on disk, after which further words are not inserted anymore (ie. 2048 is 2MB; the size should be a multiple of 1024, eg. 8 * 1024 = 8192 for 8MB; use this limit to prevent heavy graphs to be consolidating forever; this limit is enforced in pair with store.fst.graph.max_words, whichever is reached first)

  • graph.max_words (type: integer, allowed: numbers, default: 250000) — Maximum number of words that can be held at the same time in the graph, after which further words are not inserted anymore (use this limit to prevent heavy graphs to be consolidating forever; this limit is enforced in pair with store.fst.graph.max_size, whichever is reached first)

Environment variables interpolation

Some configuration keys —namely server.log_level, channel.inet, channel.auth_password, store.kv.path and store.fst.path— support environment variable interpolation. If you set "${env.SECRET}" for one of those keys, the value will be expanded from the SECRET environment variable.