manual/english/Changelog.md
Released: March 30th 2026
This release brings a broader packaging overhaul, new API_URL and API_TIMEOUT options for auto-embedding models, hybrid search, S3-compatible backup and restore support, faster RT tables maintenance with N-way merges and parallel OPTIMIZE, KNN prefiltering, and MySQL-compatible prepared statements.
API_URL and API_TIMEOUT for auto-embedding models. This affects you if you use auto-embedding models or manage MCL separately from the daemon, because Manticore 25.0.0 requires MCL 13.0.0 and is not compatible with older MCL versions; upgrade the daemon and MCL together, and downgrade is only safe if you also restore a matching older MCL version.bigram_delimiter modes and digit-aware bigram_index modes. This affects you if you use those bigram options or if you rebuild indexes and later need to open the rewritten indexes with an older Manticore version: existing older indexes remain readable, but indexes rebuilt or newly written in format 69 are not compatible with older Manticore versions, so downgrade is only safe before such indexes are rewritten.model_name: it preserves user-provided vectors, prevents replicas from regenerating different embeddings, and improves auto-embeddings performance when used with replication, but mixed-version replication clusters are not compatible across this change. Upgrade clustered nodes together; downgrade is only safe before the new replication protocol is used in the cluster.hitless_words dictionary layout handling. This affects you if you rebuild or newly write indexes and later need to open them with an older Manticore version: existing older indexes remain readable, but any indexes rebuilt or newly written in the new format are not compatible with older Manticore versions, so downgrade is only safe before such indexes are rewritten.manticore is now a bundle package for deb and rpm. It includes the daemon, tools, converter, development headers, ICU data, bundled dependency packages (manticore-columnar-lib, manticore-backup, manticore-buddy, manticore-executor, manticore-load, manticore-galera, and manticore-tzdata where applicable), plus built-in language packs for German, English, and Russian, along with Jieba support. The Ukrainian lemmatizer is not bundled; install it separately using the Debian/Ubuntu or RHEL/CentOS instructions.
When upgrading from the previous package layout, normally you can just install the manticore package. If the old split packages cause conflicts, remove them with apt remove 'manticore*' or yum remove 'manticore*' and then install manticore. That cleanup does not remove your existing data or configuration.OPTIMIZE jobs, reducing the time needed to merge many disk chunks and exposing the new merge_chunks_per_job and parallel_chunk_merges settings.fusion_method='rrf'), allowing SQL and JSON queries to combine full-text and KNN results, rank them with hybrid_score(), and support multiple KNN subqueries in a single fused result set.interactive_timeout and wait_timeout handling for SQL connections, so interactive clients and non-interactive clients can use different idle timeouts like MySQL.max_query_time interrupts queries using operators such as NOTNEAR, MAYBE, or negation, so timed-out searches no longer return rows that do not actually satisfy the query.SHOW STATUS timings stay consistent, and Buddy’s internal requests no longer distort user-facing daemon stats.searchd startup on macOS so Buddy-backed SQL commands no longer fail from loading an older system libcurl; runtime lookup now prefers Homebrew curl automatically, with MANTICORE_CURL_LIB still available as an explicit override.searchd links against an incompatible libcurl; the package now prefers Homebrew curl and supports a MANTICORE_CURL_LIB runtime override.UPDATE handling when a changed attribute must also disable its secondary index, preventing incorrect warnings and inconsistent secondary-index state.html_strip=1.float_vector columns to support an explicit empty FROM='', which now correctly means “use all text/string fields.”.float_vector columns so MVA values are preserved instead of being stored as empty.IMPORT TABLE to copy external files correctly, including hitless_words, so imported tables no longer reference missing external files after import.LEFT JOIN results to emit proper MySQL NULL values instead of the string NULL, improving compatibility with native MySQL clients and drivers.WHERE title='test'; these queries now return an error instead of crashing./sql?mode=raw multi-statement requests using .@files followed by SHOW META.ALTER TABLE ... ADD COLUMN for auto-embedding float_vector columns so model_name and FROM are preserved correctly instead of being replaced with knn_dims='0'.Resource temporarily unavailable during request processing.AND nodes, improving query planner decisions for intersecting full-text conditions.indextool --check failing to open disk-chunk hitless_words files by resolving their paths relative to the index directory.LEFT JOIN queries after flushing RAM chunks to disk.search_stats_ms_* metrics overreporting search time by double-counting local execution time; these statistics now match the actual query execution time.HIGHLIGHT() and snippet generation for CJK text adding unnecessary spaces between words; highlighted excerpts now preserve the original text spacing instead of tokenized spacing.TRUNCATE TABLE ... WITH RECONFIGURE for RT tables with index_field_lengths=1, particularly when fields and string attributes are mixed; field-length attributes are now set up correctly during reconfigure.SELECT t2.i al, (al*0.1) pct ... WHERE pct > 0, which could fail with incoming-schema expression missing evaluator; these filters now evaluate correctly after the join.Released: February 7th 2026
If you follow the official installation guide, you don't need to worry about this.
❤️ We’d like to thank @pakud for their work on PR #4075.
DROP CACHE. This updates the interface between the daemon and MCL. Older Manticore Search versions don't suport the newer MCL._id and _score as numbers instead of strings, matching regular search; this is a breaking change for clients that relied on string type for these fields.CREATE TABLE (semicolon-separated values; wordforms/exceptions use > or => for pairs, with \ escaping), so tables can be created without external files; SHOW CREATE TABLE returns these inline values.SELECT * with KNN on columnar tables.--quiet (-q) flag to suppress startup output (banner and precache messages), printing only errors; useful when starting and stopping searchd in a loop or from scripts.Keep-Alive header explicitly, reducing random connection failures in API clients (e.g. PHP, Go). To close a connection, the client sends Connection: close. HTTP/1.0 still requires Connection: keep-alive for persistence.j.s) returning no matches; they now work like joins on plain string attributes.REPLACE INTO as well as INSERT INTO, so tables are created on demand when missing.GROUP BY results on columnar MVA with multiple GROUP BY columns by rejecting such queries with the same error as rowwise ("MVA values can't be used in multiple group-by").highlight() with html_strip_mode=strip corrupting content by decoding entities and altering tags; original entity form is now preserved in the highlighted output.ALTER TABLE REBUILD SECONDARY failing with failed to rename … .tmp.spjidx when the table had multiple disk chunks.ATTACH TABLE now migrates these files to the new chunk-based format and updates disk chunk headers, so after ALTER TABLE RENAME the daemon no longer reports missing external files on restart.(@name ="^New York$") | (@ascii_name ="^New York$")) returning matches from other fields; boolean simplification no longer drops field restrictions in that case.ALTER TABLE with table-level settings (e.g. html_strip='1') failing on tables with auto-embeddings with "knn_dims can't be used together with model_name"; serialization now omits knn_dims when model_name is set.HIGHLIGHT and snippet generation now support REGEX in the query string, so REGEX matches are highlighted correctly when the same REGEX query is used in MATCH and HIGHLIGHT.HISTOGRAM() returning incorrect values (e.g. zero) when the first argument is a float expression such as price*100; histogram now uses float bucketing for float-like arguments so expression results are binned correctly.HIGHLIGHT() and snippet generation now support the explicit OR operator (|) inside quoted expressions, using the same query transform as search.DROP CACHE to clear query, docstore, skip, and secondary index caches.ORDER BY included a string column: the scroll token is now applied correctly and returns the next page instead of repeating the first page.WHERE clause filters; previously in some cases (e.g. when the right table had no full-text field) the query ran without error and returned an empty result set.CREATE TABLE and in expressions, so tables and queries can use reserved words as identifiers when escaped with backticks.SHOW CREATE TABLE now outputs column names that are reserved or special (e.g. knn) in backticks, so that CREATE TABLE ... LIKE and re-executing the shown DDL work when the source table has such columns.ALTER so the non-existent .spds rename no longer occurs, and preserved existing blob data in the blob pool when adding columnar attributes so "Blob offset out of bounds" and query/merge crashes no longer occur.0000000000), producing invalid JSON; string attributes are now always serialized as quoted JSON strings.searchd.attr_autoconv_strict option (default 0) to control string-to-number conversion on INSERT/REPLACE into RT tables: when enabled, invalid values (empty string, non-numeric, trailing characters, overflow) return errors instead of silently converting to 0.must_not in JSON search so it behaves as logical NOT (returning all documents that do not match the inner query); also fixed SphinxQL WHERE NOT ... for filter expressions.libstemmer_el) bug #204, preventing the server crash that could occur with certain Greek text when using morphology='libstemmer_el' (see issue #2888).LazyNetEvents_c::EventTick().CREATE TABLE, so either naming can be used when defining tables (e.g. CREATE TABLE t (id bigint, tags mva64, ...) type='pq'). Schema output (e.g. DESC table) continues to use the existing type names for compatibility.ALTER CLUSTER ... ADD TABLE was run on another node after the joiner had restarted and re-joined.NOTNEAR full-text operator so it correctly excludes matches when the right-hand term appears within the specified distance either before or after the left-hand term (symmetric/backward matching). Previously NOTNEAR only considered the right term when it appeared after the left term, so for example d NOTNEAR/3 a could incorrectly match a document containing "a b c d". NEAR now also returns an error when the distance is 0.OPTION fuzzy=1) followed by show meta on /cli and /sql?mode=raw.Released: December 7th 2025
If you follow the official installation guide, you don't need to worry about this.
Content-Type so /metrics now returns Prometheus text format (text/plain; version=0.0.4) instead of JSON, fixing scraping failures.LOCK TABLES / UNLOCK TABLES: write locks now return warnings instead of errors, read locks are correctly shown in SHOW LOCKS, and the overall lock logic behaves consistently.JOIN ON clauses (not just equality comparisons), enabling queries like ... ON t1.id = t2.t1_id AND t2.value = 5./sql responses from the daemon incorrectly used the Content-Type: text/html header instead of application/json.GROUP BY / FACET on attributes generated with a JSON-to-attribute mapping failed to work./sql endpoint and includes other fuzzy-search fixes./cli_json that include semicolons (e.g. combining SQL statements) failed — semicolons are no longer replaced with null characters before processing.persistent_connections_limit.query_time_1min) actually reported milliseconds instead of seconds.SHOW INDEX <name> STATUS search timing stats (search_stats_ms_*) did not match the values calculated from the query logs; the reported timings now reflect the real logged query times.application/json.binlog.cpp, ensuring DoSaveMeta() now compiles correctly for Windows platforms.MATCH() parsing so empty groups like () no longer trigger an error (e.g. queries such as camera() now work correctly).--skip-lock-tables when dumping replicated tables, and fixed several broken manual links.LOCK/UNLOCK TABLES checks (now handled by the daemon) and fixing autocomplete parsing so queries with escaped quotes (e.g. \" or \') are handled correctly instead of failing.knn_dist() in a custom ranker expression with KNN+MATCH queries; such queries now return a clear error instead of crashing.Released: November 7th 2025
❤️ We’d like to extend our heartfelt thanks to @ricardopintottrdata for their work on PR #3792 and PR #3828 — addressing issues around HAVING total counts and the filter with empty name error — and to @jdelStrother for their contribution with PR #3819, which improves handling of ParseCJKSegmentation when Jieba support isn’t available.
Your efforts help make the project stronger — many thanks!
If you follow the official installation guide, you don't need to worry about this.
Version v14.0.0 updates the replication protocol. If you are running a replication cluster, you need to:
--new-cluster, using the tool manticore_new_cluster in Linux.LOCK TABLES statements generated by mysqldump, improving the safety of logical backups.max(ft field)."(abc|def)" didn't work as expected.HAVING.CALL SUGGEST not matching trigrams.ParseCJKSegmentation when Jieba support is not available.Released: October 7th 2025
If you follow the official installation guide, you don't need to worry about this.
Since the configuration file was updated, you might see a warning during the upgrade on Linux asking whether to keep your version or use the new one from the package. If you have a custom (non-default) configuration, it's recommended to keep your version and update the pid_file path to /run/manticore/searchd.pid. However, it should still work normally even if you don't change the path.
/var/run/manticore with the correct /run/manticore in the configuration. Since the configuration file was updated, you may see a warning during upgrade asking whether to keep your version or use the new one from the package. If you have a custom (non-default) configuration, it's recommended to keep your version and update the pid_file path to /run/manticore/searchd.pid.Released: September 13th 2025
Released: September 13th 2025
The main highlight of this release is Auto Embeddings — a new feature that makes semantic search as easy as SQL. No need for external services or complex pipelines: just insert text and search with natural language.
If you follow the official installation guide, you don't need to worry about this.
Recommended MCL version: 8.0.1 Recommended Buddy version: 3.34.2
/run.boolean_simplify in the configuration.alter table <table> modify column <column> api_key=<key>./sql endpoint no longer allows the SHOW VERSION command.Released: August 8th 2025
Recommended MCL version: 8.0.1 Recommended Buddy version: 3.34.2
If you follow the official installation guide, you don't need to worry about this.
Released: July 8th 2025
@@collation_database that caused incompatibility with certain mysqldump versionsSHOW THREADS to display CPU utilization as an integerdist/test_kit_docker_build.sh that caused the Buddy commit to be missingReleased: June 9th 2025
This version represents an update with new features, one breaking change and numerous stability improvements and bug fixes. The changes focus on enhancing monitoring capabilities, improving search functionality, and fixing various critical issues that affected system stability and performance.
Starting with version 10.1.0, CentOS 7 is no longer supported. Users are encouraged to upgrade to a supported operating system.
layouts='' as the default for fuzzy searchseqno periodically for faster node restart after crash@@variables always returned 0SELECT ... FUZZY=0 didn't always disable fuzzy search_random sorting in JSON APIid != value%20 and +)json.field in facet searchDELETE FROM with TRUNCATE for distributed tablesgeodist() with JSON attributesReleased: May 2nd 2025
This release includes multiple bug fixes and stability improvements, better table usage tracking, and enhancements to memory and resource management.
❤️ Special thanks to @cho-m for fixing build compatibility with Boost 1.88.0 and @benwills for improving documentation around stored_only_fields.
🪲 v9.3.2 Fixed an issue where the "Show Threads" column showed CPU activity as a float instead of a string; also fixed a PyMySQL result set parsing error caused by a wrong data type.
🪲 v9.3.1 Issue #3343 Fixed leftover tmp.spidx files when the optimize process was interrupted.
🆕 v9.3.0 PR #3337 Added a per-table command counter and detailed table usage statistics.
🪲 v9.2.39 Issue #3236 Fix: prevent table damage by removing complex chunk updates. Using wait functions inside the serial worker broke serial processing, which could damage tables. Reimplemented autoflush. Removed external polling queue to avoid unnecessary table locks. Added "small table" condition: if the number of documents is below the 'small table limit' (8192) and no Secondary Index (SI) is used, flushing is skipped.
🪲 v9.2.38 Fix: skip creating Secondary Index (SI) for filters using ALL/ANY on string lists, without affecting JSON attributes.
🪲 v9.2.37 Issue #2898 Added backtick support for system tables.
🪲 v9.2.36 Fix: use a placeholder for cluster operations in legacy code. In the parser, we now clearly separate fields for table and cluster names.
🪲 v9.2.35 Fix: crash when unquoting a single '.
🪲 v9.2.34 Issue #3090 Fix: handling of large document IDs (could previously fail to find them).
🪲 v9.2.33 Fix: use unsigned integers for bit vector sizes.
🪲 v9.2.32 Fix: reduce peak memory usage during merge. docid-to-rowid lookups now use 12 bytes per document instead of 16 bytes. Example: 24 GB RAM for 2 billion docs instead of 36 GB.
🪲 v9.2.31 Issue #3238 Fix: incorrect COUNT(*) value in large real-time tables.
🪲 v9.2.30 Fix: undefined behavior when zeroing string attributes.
🪲 v9.2.29 Minor fix: improved a warning text.
🪲 v9.2.28 Issue #3290 Improvement: enhanced indextool --buildidf
🪲 v9.2.27 Issue #3032 With the Kafka integration, you can now create a source for a specific Kafka partition.
🪲 v9.2.26 Issue #3301 Fix: ORDER BY and WHERE on id could cause OOM (Out Of Memory) errors.
🪲 v9.2.25 Issue #3171 Fix: crash caused by segmentation fault when using grouper with multiple JSON attributes on an RT table that has multiple disk chunks
🪲 v9.2.24 Issue #3246 Fix: WHERE string ANY(...) queries failed after RAM chunk flush.
🪲 v9.2.22 Issue #2763 Fix: global idf file was not loaded when using ALTER TABLE.
🪲 v9.2.21 Fix: global idf files can be large. We now release tables sooner to avoid holding unnecessary resources.
🪲 v9.2.20 PR #3277 Improvement: better validatation of sharding options.
🪲 v9.2.19 PR #3275 Fix: build compatibility with Boost 1.88.0.
🪲 v9.2.18 Issue #3228 Fix: crash during distributed table creation (invalid pointer issue).
🪲 v9.2.16 Issue #3063 Fix: bug in distance calculation when using the GEODIST function.
🪲 v9.2.15 Issue #3027 Minor improvement: support for Elastic's query_string filter format.
Released: March 28th 2025
--mockstack flag to calculate recursive operation stack requirements. The new --mockstack mode analyzes and reports the necessary stack sizes for recursive expression evaluation, pttern matching operations, filter processing. Calculated stack requirements are output to console for debugging and optimization purposes.searchd.kibana_version_string, which can be useful when using Manticore with specific versions of Kibana or OpenSearch Dashboards that expect a particular Elasticsearch version._id to id in some HTTP JSON responses for consistency. Make sure to update your app accordingly.server_id during cluster joins to ensure each node has a unique ID. The JOIN CLUSTER operation can now fail with an error message indicating a duplicate server_id when the joining node has the same server_id as an existing node in the cluster. To resolve this issue, ensure that each node in the replication cluster has a unique server_id. You can change the default server_id in the "searchd" section of your configuration file to a unique value before attempting to join the cluster. This change updates the replication protocol. If you are running a replication cluster, you need to:
--new-cluster, using the tool manticore_new_cluster in Linux.serializer are properly restored.ORDER BY clause.lower_bound call error with const knn::DocDist_t*&. ❤️ Thank you @Azq2 for the PR.ALTER: float vectors now keep their original dimensions, and KNN indexes are now properly generated.fuzzy=1 option couldn't be used with ranker or field_weights.SET GLOBAL timezone had no effect.UPDATE statements now correctly respect the query_log_min_msec setting.JOIN CLUSTER to fail.Released: February 28th 2025
avg() in a SELECT ... JOIN query could lead to incorrect results; this is now fixed.IN(...) could produce incorrect results.max_iops / max_iosize in version 7.0.0 could degrade indexing performance; this is now fixed.diskchunk_flush_write_timeout=-1 per table did not disable index flushing; this is now fixed.NOT operator and an expression ranker.Released: January 30th 2025
global_idf in RT tables. Requires table recreation.cjk charset. Update your charset definitions accordingly: if you have cjk,non_cjk and Thai characters are important for you, change it to cjk,thai,non_cjk, or cont,non_cjk, where cont is the new designation for all continuous-script languages (i.e., cjk + thai). Modify existing tables using ALTER TABLE.Name to Variable name for PQ SHOW META.create table / alter table. You need to make a clean shutdown of the Manticore instance before upgrading to the new version.--new-cluster, using the tool manticore_new_cluster in Linux.ALTER CLUSTER ADD and DROP. This change also affects the replication protocol. Refer to the previous section for guidance on handling this update.COUNT(DISTINCT) support for ORDER BY in FACET and GROUP BY.Content-Encoding: gzip.SHOW LOCKS command.mysqldump for replicated tables.CREATE TABLE and ALTER TABLE statements.index with table in requests and responses.distinct column to HTTP /sql endpoint aggregation results.uuid_short expression in the select list.manticore-buddy wrapper.searchd.CALL uuid_short statement to generate sequences with multiple uuid_short values.GROUP BY in SphinxQL.fixed_interval in Kibana date-related requests.searchd log in non-debug mode.json.string in the right table on table join.null for all values in any POST HTTP JSON endpoints (insert/replace/bulk). In this case a default value is used._update request.thread_stack.unattended-upgrades.CREATE TABLE and ALTER TABLE. ❤️ Thank you, @subnix for the PR.OPTIMIZE now throws an error when the table is frozen.searchd hangs on stop after FREEZE and FLUSH RAMCHUNK.FACET with more than 5 sort fields.mysqldump with index_field_lengths enabled.ALTER TABLE command.FACET query; set default sort order to DESC for FACET with COUNT(*)./sql and /sql?mode=raw; made requests from these endpoints consistent without requiring query= header.OR with KNN search.indextool --mergeidf *.idf --out global.idf deletes the output file after creation.ORDER BY string in outer select.lemmatize_xxx_all tokenizers increase hitpos of subsequent tokens.ALTER ... ADD COLUMN ... TEXT.SELECT queries to wait until the table is unfrozen._bulk requests._bulk endpoint.SELECT queries.ALL or ANY argument in IN expression is not a JSON attribute.libstemmer. ❤️ Thank you, @subnix for the PR.CREATE TABLE IF NOT EXISTS ... WITH DATA failure when the table already exists.REPLACE INTO cluster_name:table_name functionality.--network=host.Released: November 22nd 2024
Version 6.3.8 continues the 6.3 series and includes only bug fixes.
threads or max_threads_per_query settings.Released: Oct 7 2024
unattended-upgrades utility, which automatically installs package updates on Debian-based systems, would incorrectly mark several Manticore packages, including manticore-galera, manticore-executor, and manticore-columnar-lib, for removal. This occurred due to dpkg mistakenly considering the virtual package manticore-extra as redundant. Changes were made to ensure unattended-upgrades no longer tries to remove essential Manticore components.Released: August 2nd 2024
Version 6.3.6 continues the 6.3 series and includes only bug fixes.
max_query_time when querying multiple indexes.Released: July 31st 2024
Version 6.3.4 continues the 6.3 series and includes only minor improvements and bug fixes.
hitless_words.manticore.json file instead of checking the entire size of the data directory.Released: June 26th 2024
Version 6.3.2 continues the 6.3 series and includes several bug fixes, some of which were discovered after the release of 6.3.0.
local_df enabled.count(*) without GROUP BY.ALTER TABLE did not take KNN options into account.manticore-tools Redhat package in version 6.3.0.CALL KEYWORDS for RT index with disk chunks./cli requests.drop table if exists t; create table t via /cli.cluster:name format in the /_bulk HTTP endpoint.Released: May 23rd 2024
.spa (scalar attrs): 256KB -> 8MB; .spb (blob attrs): 256KB -> 8MB; .spc (columnar attrs): 1MB, no change; .spds (docstore): 256KB -> 8MB; .spidx (secondary indexes): 256KB buffer -> 128MB memory limit; .spi (dictionary): 256KB -> 16MB; .spd (doclists): 8MB, no change; .spp (hitlists): 8MB, no change; .spe (skiplists): 256KB -> 8MB.libgalera_smm.so from MySQL 5.x)._rate to _rps.index to table in error messages; fixed bison parser error message fixup.manticore.tbl as table name.agent_connect_timeout and agent_query_timeout) for create distributed table statement.searchd.expansion_limit.SHOW STATUS.@@system.sessions./sql endpoint to ease integration with libraries.CALL PQ with large packets.log_level=debug is set.DEBUG DEDUP command for real-time table chunks that can experience duplicate entries after attaching a plain table containing duplicates.@timestamp column as timestamp.RuntimeDirectory.local_df is now a default. Improved the master-agent search protocol (version updated). If you are running Manticore Search in a distributed environment with multiple instances, make sure to first upgrade the agents, then the masters.--new-cluster, using the tool manticore_new_cluster in Linux./json/* have been deprecated.plugin_dir anymore./search responses.charset_table was not working in some cases.gcache.page_size for replication clusters without tables or with empty tables; also fixed saving and loading of the Galera options.searchd.agent_* but with different defaults.show variables.count distinct which could cause a crash.index_token_filter.debugv verbosity level.index_exact_words between indexing and loading the table to the daemon.=term of full-text query with the morphology_skip_fields field.data_dir affect the current work directory on daemon start.agent_query_timeout being replaced by the default query option agent_query_timeout.packedfactors() with multiple values per match.min_prefix_len / min_infix_len.SPH_EXTNODE_STACK_SIZE value.FACET error when querying a distributed table with agent and local tables.WHERE json.array IN (<value>)./cli.CREATE TABLE wasn't failing in case of a missing wordforms file.SPH_SORT_ATTR_DESC and SPH_SORT_ATTR_ASC.Expect: 100-continue HTTP header for curl requests to Buddy./search.ALTER CLUSTER ADD and JOIN CLUSTER operations to wait for each other, preventing a race condition where ALTER adds a table to the cluster while the donor sends tables to the joiner node./pq/{table}/* requests.UNFREEZE wasn't working in some cases./cli_json requests.signal 11 when inserting data.FREEZE counter to avoid freeze/unfreeze issues.max_query_time could be not working in some cases.SecondaryIndex CBO hint.expansion_limit to slice final result set for call keywords from multiple disk chunks or RAM chunks.alter failed when the data_dir path was located on a symlink.--force flagtmp.spidx files.HAVING.expr ranker on using columnar attribute.manticore.conf.sh.Released: August 23rd 2023
Version 6.2.12 continues the 6.2 series and addresses issues discovered after the release of 6.2.0.
TimeoutStartSec from infinity to 0 for better compatibility with Centos 7.searchdreplication.cpp: beggining -> beginning.Thd_t build issue on Windows related to atomic copy restrictions.ColumnarScan.AF_INET error in the test./bulk endpoints in the manual.Released: August 4th 2023
mysqldumppseudo_sharding has been adjusted to be limited to the number of free threads. This update considerably enhances the throughput performance./json/pq HTTP endpoint.upper() and lower().count(*) queries, a precalculated value is now returned.SELECT for making arbitrary calculations and displaying @@sysvars. Unlike before, you are no longer limited to just one calculation. Therefore, queries like select user(), database(), @@version_comment, version(), 1+1 as a limit 10 will return all the columns. Note that the optional 'limit' will always be ignored.CREATE DATABASE stub query.ALTER TABLE table REBUILD SECONDARY, secondary indexes are now always rebuilt, even if attributes weren't updated.SELECT DATABASE() command. However, it will always return Manticore. This addition is crucial for integrations with various MySQL tools./cli_json endpoint to function as the previous /cli.thread_stack can now be altered during runtime using the SET statement. Both session-local and daemon-wide variants are available. Current values can be accessed in the show variables output.SHOW STATUS command.DESC and SHOW CREATE TABLE now match that of SELECT * FROM.P01) during various errors. This enhancement aids in identifying which parser caused an error and also obscures non-essential internal details.sentence to show the entire sentencestrftime() function./bulk endpoint reports information regarding the number of processed and non-processed strings (documents) in case of an error.CREATE TABLE operation can run at a time.Get call, replacing the previous two-step AdvanceTo + Get calls to retrieve a value.CheckReplaceEntry call was removed from the group sorter to expedite the calculation of aggregate functions.CREATE TABLE options read_buffer_docs and read_buffer_hits now support k/m/g syntax.apt/yum install manticore-language-packs. On macOS, use the command brew install manticoresoftware/tap/manticore-language-packs.SHOW CREATE TABLE and DESC operations.INSERT queries, new INSERT queries will fail until enough disk space becomes available./bulk endpoint now processes empty lines as a commit command. More info here.count(*) is used with a single filter, queries now leverage precalculated data from secondary indexes when available, substantially speeding up query times./*+ SecondaryIndex(uid) */. Please note that the old syntax is no longer supported.@ in table names has been disallowed to prevent syntax conflicts.indexed and attribute are now regarded as a single field during INSERT, DESC, and ALTER operations.manticore.json config..sph files could be corrupted ALTER. Fixed.pre_commit error occurring when replace is replicated from multiple master nodes.pseudo_sharding was disabled.show index status command has been modified and now varies depending on the type of index in use.expand_keywords option.SNIPPETS() was called.not_terms_only_allowed option to RT index with killed documents.FEDERATED engine with aggregate.rt_attr_json column was incompatible with columnar storage.ignore_chars.--dumpdocids command.morphology_skip_fields.max_packet_size check for replication commands between nodes. Additionally, the latest cluster error has been added to the status display.MANTICORE_BUDDY_TIMEOUT (default 3 seconds) to control the daemon's wait duration for a buddy message at startup.SHOW CREATE TABLE.SNIPPET() function.all()/any() is logged.Released: March 15 2023
/pq HTTP endpoint to be an alias of the /json/pq HTTP endpoint.Released: Feb 10 2023
Released: Feb 7 2023
Starting with this release, Manticore Search comes with Manticore Buddy, a sidecar daemon written in PHP that handles high-level functionality that does not require super low latency or high throughput. Manticore Buddy operates behind the scenes, and you may not even realize it is running. Although it is invisible to the end user, it was a significant challenge to make Manticore Buddy easily installable and compatible with the main C++-based daemon. This major change will allow the team to develop a wide range of new high-level features, such as shards orchestration, access control and authentication, and various integrations like mysqldump, DBeaver, Grafana mysql connector. For now it already handles SHOW QUERIES, BACKUP and Auto schema.
This release also includes more than 130 bug fixes and numerous features, many of which can be considered major.
SET GLOBAL ES_COMPAT=off.manticore-backup for backing up and restoring Manticore instanceKILL to kill a long-running SELECT.max_matches for aggregation queries to increase accuracy and lower response time.Issue #822 SQL commands FREEZE/UNFREEZE to prepare a real-time/plain table for a backup.
Commit c470 New settings accurate_aggregation and max_matches_increase_threshold for controlled aggregation accuracy.
Issue #718 Support for signed negative 64-bit IDs. Note, you still can't use IDs > 2^63, but you can now use ids in the range of from -2^63 to 0.
As we recently added support for secondary indexes, things became confusing as "index" could refer to a secondary index, a full-text index, or a plain/real-time index. To reduce confusion, we are renaming the latter to "table". The following SQL/command line commands are affected by this change. Their old versions are deprecated, but still functional:
index <table name> => table <table name>,searchd -i / --index => searchd -t / --table,SHOW INDEX STATUS => SHOW TABLE STATUS,SHOW INDEX SETTINGS => SHOW TABLE SETTINGS,FLUSH RTINDEX => FLUSH TABLE,OPTIMIZE INDEX => OPTIMIZE TABLE,ATTACH TABLE plain TO RTINDEX rt => ATTACH TABLE plain TO TABLE rt,RELOAD INDEX => RELOAD TABLE,RELOAD INDEXES => RELOAD TABLES.We are not planning to make the old forms obsolete, but to ensure compatibility with the documentation, we recommend changing the names in your application. What will be changed in a future release is the "index" to "table" rename in the output of various SQL and JSON commands.
Queries with stateful UDFs are now forced to be executed in a single thread.
Issue #1011 Refactoring of all related to time scheduling as a prerequisite for parallel chunks merging.
⚠️ BREAKING CHANGE: Columnar storage format has been changed. You need to rebuild those tables that have columnar attributes.
⚠️ BREAKING CHANGE: Secondary indexes file format has been changed, so if you are using secondary indexes for searching and have searchd.secondary_indexes = 1 in your configuration file, be aware that the new Manticore version will skip loading the tables that have secondary indexes. It's recommended to:
searchd.secondary_indexes to 0 in the configuration file.ALTER TABLE <table name> REBUILD SECONDARY for each index to rebuild secondary indexes.If you are running a replication cluster, you'll need to run ALTER TABLE <table name> REBUILD SECONDARY on all the nodes or follow this instruction with just change: run the ALTER .. REBUILD SECONDARY instead of the OPTIMIZE.
⚠️ BREAKING CHANGE: The binlog version has been updated, so any binlogs from previous versions will not be replayed. It is important to ensure that Manticore Search is stopped cleanly during the upgrade process. This means that there should be no binlog files in /var/lib/manticore/binlog/ except for binlog.meta after stopping the previous instance.
Issue #849 SHOW SETTINGS: you can now see the settings from the configuration file from inside Manticore.
Issue #1007 SET GLOBAL CPUSTATS=1/0 turns on/off cpu time tracking; SHOW THREADS now doesn't show CPU statistics when the cpu time tracking is off.
Issue #1009 RT table RAM chunk segments can now be merged while the RAM chunk is being flushed.
Issue #1012 Added secondary index progress to the output of indexer.
Issue #1013 Previously a table record could be removed by Manticore from the index list if it couldn't start serving it on start. The new behaviour is to keep it in the list to try to load it on the next start.
indextool --docextract returns all the words and hits belonging to requested document.
Commit 2b29 Environment variable dump_corrupt_meta enables dumping a corrupted table meta data to log in case searchd can't load the index.
Commit c7a3 DEBUG META can show max_matches and pseudo sharding statistics.
Commit 6bca A better error instead of the confusing "Index header format is not json, will try it as binary...".
Commit bef3 Ukirainian lemmatizer path has been changed.
Commit 4ae7 Secondary indexes statistics has been added to SHOW META.
Commit 2e7c JSON interface can now be easily visualized using Swagger Editor https://manual.manticoresearch.com/Openapi#OpenAPI-specification.
⚠️ BREAKING CHANGE: Replication protocol has been changed. If you are running a replication cluster, then when upgrading to Manticore 5 you need to:
--new-cluster (run tool manticore_new_cluster in Linux).select attr, count(*) from plain_index (w/o filtering) are now faster in case you are using MCL.brew install manticoresoftware/manticore/manticoresearch manticoresoftware/manticore/manticore-extra.textbinlog_flush = 1 has been broken all the time since Sphinx. Fixed.got exception while reading ist stream: mkstemp(./gmb_pF6TJi) failed: 13 (Permission denied) if the searchd was started from a directory it can't write to.Released: May 30th 2022
Released: May 18th 2022
🔬 Support for Manticore Columnar Library 1.15.2, which enables Secondary indexes beta version. Building secondary indexes is on by default for plain and real-time columnar and row-wise indexes (if Manticore Columnar Library is in use), but to enable it for searching you need to set secondary_indexes = 1 either in your configuration file or using SET GLOBAL. The new functionality is supported in all operating systems except old Debian Stretch and Ubuntu Xenial.
Read-only mode: you can now specify listeners that process only read queries discarding any writes.
New /cli endpoint for running SQL queries over HTTP even easier.
Faster bulk INSERT/REPLACE/DELETE via JSON over HTTP: previously you could provide multiple write commands via HTTP JSON protocol, but they were processed one by one, now they are handled as a single transaction.
#720 Nested filters support in JSON protocol. Previously you couldn't code things like a=1 and (b=2 or c=3) in JSON: must (AND), should (OR) and must_not (NOT) worked only on the highest level. Now they can be nested.
Support for Chunked transfer encoding in the HTTP protocol. You can now use chunked transfer in your application to transmit large batches with reduced resource consumption (since calculating Content-Length is unnecessary). On the server side, Manticore now always processes incoming HTTP data in a streaming manner, without waiting for the entire batch to be transferred as before, which:
max_packet_size (128MB), for example, 1GB at a time.#719 HTTP interface support of 100 Continue: now you can transfer large batches from curl (including curl libraries used by various programming languages) which by default does Expect: 100-continue and waits some time before actually sending the batch. Previously you had to add Expect: header, now it's not needed.
Previously (note the response time):
$ time curl -v -sX POST http://localhost:9318/bulk -H "Content-Type: application/x-ndjson" --data '{"insert": {"index": "user", "doc": {"name":"Prof. Matt Heaney IV","email":"[email protected]","description":"Tempora ullam eaque consequatur. Vero aut minima ut et ut omnis officiis vel. Molestiae quis voluptatum sint numquam.","age":15,"active":1}}}
{"insert": {"index": "user", "doc": {"name":"Prof. Boyd McKenzie","email":"[email protected]","description":"Blanditiis maiores odio corporis eaque illum. Aut et rerum iste. Neque et ullam quisquam officia dignissimos quo cumque.","age":84,"active":1}}}
{"insert": {"index": "user", "doc": {"name":"Mr. Johann Smith","email":"[email protected]","description":"Temporibus amet magnam consequatur omnis consequatur illo fugit. Debitis natus doloremque est tempore deserunt vero. Harum eos corrupti nemo ut.","age":89,"active":1}}}
{"insert": {"index": "user", "doc": {"name":"Hector Pouros","email":"[email protected]","description":" as voluptatem inventore sit. Aliquam fugit perferendis est id aut odio et sapiente.","age":64,"active":1}}}'
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9318 (#0)
> POST /bulk HTTP/1.1
> Host: localhost:9318
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/x-ndjson
> Content-Length: 1025
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Server: 4.2.0 15e927b@211223 release (columnar 1.11.4 327b3d4@211223)
< Content-Type: application/json; charset=UTF-8
< Content-Length: 434
<
* Connection #0 to host localhost left intact
{"items":[{"insert":{"table":"user","_id":2811798918248005633,"created":true,"result":"created","status":201}},{"insert":{"table":"user","_id":2811798918248005634,"created":true,"result":"created","status":201}},{"insert":{"table":"user","_id":2811798918248005635,"created":true,"result":"created","status":201}},{"insert":{"table":"user","_id":2811798918248005636,"created":true,"result":"created","status":201}}],"errors":false}
real 0m1.022s
user 0m0.001s
sys 0m0.010s
Now:
$ time curl -v -sX POST http://localhost:9318/bulk -H "Content-Type: application/x-ndjson" --data '{"insert": {"index": "user", "doc": {"name":"Prof. Matt Heaney IV","email":"[email protected]","description":"Tempora ullam eaque consequatur. Vero aut minima ut et ut omnis officiis vel. Molestiae quis voluptatum sint numquam.","age":15,"active":1}}}
{"insert": {"index": "user", "doc": {"name":"Prof. Boyd McKenzie","email":"[email protected]","description":"Blanditiis maiores odio corporis eaque illum. Aut et rerum iste. Neque et ullam quisquam officia dignissimos quo cumque.","age":84,"active":1}}}
{"insert": {"index": "user", "doc": {"name":"Mr. Johann Smith","email":"[email protected]","description":"Temporibus amet magnam consequatur omnis consequatur illo fugit. Debitis natus doloremque est tempore deserunt vero. Harum eos corrupti nemo ut.","age":89,"active":1}}}
{"insert": {"index": "user", "doc": {"name":"Hector Pouros","email":"[email protected]","description":" as voluptatem inventore sit. Aliquam fugit perferendis est id aut odio et sapiente.","age":64,"active":1}}}'
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9318 (#0)
> POST /bulk HTTP/1.1
> Host: localhost:9318
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/x-ndjson
> Content-Length: 1025
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Server: 4.2.1 63e5749@220405 dev
< Content-Type: application/json; charset=UTF-8
< Content-Length: 0
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Server: 4.2.1 63e5749@220405 dev
< Content-Type: application/json; charset=UTF-8
< Content-Length: 147
<
* Connection #0 to host localhost left intact
{"items":[{"bulk":{"table":"user","_id":2811798919590182916,"created":4,"deleted":0,"updated":0,"result":"created","status":201}}],"errors":false}
real 0m0.015s
user 0m0.005s
sys 0m0.004s
⚠️ BREAKING CHANGE: Pseudo sharding is enabled by default. If you want to disable it make sure you add pseudo_sharding = 0 to section searchd of your Manticore configuration file.
Having at least one full-text field in a real-time/plain index is not mandatory anymore. You can now use Manticore even in cases not having anything to do with full-text search.
Fast fetching for attributes backed by Manticore Columnar Library: queries like select * from <columnar table> are now much faster than previously, especially if there are many fields in the schema.
⚠️ BREAKING CHANGE: Implicit cutoff. Manticore now doesn't spend time and resources processing data you don't need in the result set which will be returned. The downside is that it affects total_found in SHOW META and hits.total in JSON output. It is now only accurate in case you see total_relation: eq while total_relation: gte means the actual number of matching documents is greater than the total_found value you've got. To retain the previous behaviour you can use search option cutoff=0, which makes total_relation always eq.
⚠️ BREAKING CHANGE: All full-text fields are now stored by default. You need to use stored_fields = (empty value) to make all fields non-stored (i.e. revert to the previous behaviour).
#715 HTTP JSON supports search options.
.meta, .sph) were in binary format, now it's just json. The new Manticore version will convert older indexes automatically, but:
WARNING: ... syntax error, unexpected TOK_IDENTSHOW META after SELECT and it will work the same way it works via mysql. Note, previously Connection: keep-alive HTTP header was supported too, but it only caused reusing the same connection. Since this version it also makes the session stateful.columnar_attrs = * to define all your attributes as columnar in the plain mode which is useful in case the list is long.--new-cluster (run tool manticore_new_cluster in Linux).127.0.0.1 instead of 0.0.0.0 in case no listen at all is specified in config. Even though in the default configuration which is shipped with Manticore Search the listen setting is specified and it's not typical to have a configuration with no listen at all, it's still possible. Previously Manticore would listen on 0.0.0.0 which is not secure, now it listens on 127.0.0.1 which is usually not exposed to the Internet.AVG() accuracy: previously Manticore used float internally for aggregations, now it uses double which increases the accuracy significantly.DEBUG malloc_stats support for jemalloc.sphinxql by default. If you are used to plain format you need to add query_log_format = plain to your configuration file.max_connections limit, which could cause "maxed out" error for non-VIP connections. Now VIP connections are not counted towards the limit. Current number of VIP connections can be also seen in SHOW STATUS and status./sql?mode=raw now requires escaping and returns an array./bulk INSERT/REPLACE/DELETE requests:
low_priority and boolean_simplify now require a value (0/1): previously you could do SELECT ... OPTION low_priority, boolean_simplify, now you need to do SELECT ... OPTION low_priority=1, boolean_simplify=1.query_log_format=sphinxql. Previously only full-text part was logged, now it's logged as is.⚠️ BREAKING CHANGE: because of the new structure when you upgrade to Manticore 5 it's recommended to remove old packages before you install the new ones:
yum remove manticore*apt remove manticore*New deb/rpm packages structure. Previous versions provided:
manticore-server with searchd (main search daemon) and all needed for itmanticore-tools with indexer and indextoolmanticore including everythingmanticore-all RPM as a meta package referring to manticore-server and manticore-toolsThe new structure is:
manticore - deb/rpm meta package which installs all the above as dependenciesmanticore-server-core - searchd and everything to run it alonemanticore-server - systemd files and other supplementary scriptsmanticore-tools - indexer, indextool and other toolsmanticore-common - default configuration file, default data directory, default stopwordsmanticore-icudata, manticore-dev, manticore-converter didn't change much.tgz bundle which includes all the packagesSupport for Ubuntu Jammy
Support for Amazon Linux 2 via YUM repo
application/x-ndjsonranker could be specified twice in query logIt takes 48 seconds to insert 1M PQ rules and 406 seconds to REPLACE just 40K in 10K batches.
root@perf3 ~ # mysql -P9306 -h0 -e "drop table if exists pq; create table pq (f text, f2 text, j json, s string) type='percolate';"; date; for m in `seq 1 1000`; do (echo -n "insert into pq (id,query,filters,tags) values "; for n in `seq 1 1000`; do echo -n "(0,'@f (cat | ( angry dog ) | (cute mouse)) @f2 def', 'j.json.language=\"en\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; [ $n != 1000 ] && echo -n ","; done; echo ";")|mysql -P9306 -h0; done; date; mysql -P9306 -h0 -e "select count(*) from pq"
Wed Dec 22 10:24:30 AM CET 2021
Wed Dec 22 10:25:18 AM CET 2021
+----------+
| count(*) |
+----------+
| 1000000 |
+----------+
root@perf3 ~ # date; (echo "begin;"; for offset in `seq 0 10000 30000`; do n=0; echo "replace into pq (id,query,filters,tags) values "; for id in `mysql -P9306 -h0 -NB -e "select id from pq limit $offset, 10000 option max_matches=1000000"`; do echo "($id,'@f (tiger | ( angry bear ) | (cute panda)) @f2 def', 'j.json.language=\"de\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; n=$((n+1)); [ $n != 10000 ] && echo -n ","; done; echo ";"; done; echo "commit;") > /tmp/replace.sql; date
Wed Dec 22 10:26:23 AM CET 2021
Wed Dec 22 10:26:27 AM CET 2021
root@perf3 ~ # time mysql -P9306 -h0 < /tmp/replace.sql
real 6m46.195s
user 0m0.035s
sys 0m0.008s
It takes 34 seconds to insert 1M PQ rules and 23 seconds to REPLACE them in 10K batches.
root@perf3 ~ # mysql -P9306 -h0 -e "drop table if exists pq; create table pq (f text, f2 text, j json, s string) type='percolate';"; date; for m in `seq 1 1000`; do (echo -n "insert into pq (id,query,filters,tags) values "; for n in `seq 1 1000`; do echo -n "(0,'@f (cat | ( angry dog ) | (cute mouse)) @f2 def', 'j.json.language=\"en\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; [ $n != 1000 ] && echo -n ","; done; echo ";")|mysql -P9306 -h0; done; date; mysql -P9306 -h0 -e "select count(*) from pq"
Wed Dec 22 10:06:38 AM CET 2021
Wed Dec 22 10:07:12 AM CET 2021
+----------+
| count(*) |
+----------+
| 1000000 |
+----------+
root@perf3 ~ # date; (echo "begin;"; for offset in `seq 0 10000 990000`; do n=0; echo "replace into pq (id,query,filters,tags) values "; for id in `mysql -P9306 -h0 -NB -e "select id from pq limit $offset, 10000 option max_matches=1000000"`; do echo "($id,'@f (tiger | ( angry bear ) | (cute panda)) @f2 def', 'j.json.language=\"de\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; n=$((n+1)); [ $n != 10000 ] && echo -n ","; done; echo ";"; done; echo "commit;") > /tmp/replace.sql; date
Wed Dec 22 10:12:31 AM CET 2021
Wed Dec 22 10:14:00 AM CET 2021
root@perf3 ~ # time mysql -P9306 -h0 < /tmp/replace.sql
real 0m23.248s
user 0m0.891s
sys 0m0.047s
searchd. It's useful when you want to limit the RT chunks count in all your indexes to a particular number globally.YEAR() and other timestamp functions.rt_mem_limit of data before saving a new disk chunk to disk, and while saving was still collecting up to 10% more (aka double-buffer) to minimize possible insert suspension. If that limit was also exhausted, adding new documents was blocked until the disk chunk was fully saved to disk. The new adaptive limit is built on the fact that we have auto-optimize now, so it's not a big deal if disk chunks do not fully respect rt_mem_limit and start flushing a disk chunk earlier. So, now we collect up to 50% of rt_mem_limit and save that as a disk chunk. Upon saving we look at the statistics (how much we've saved, how many new documents have arrived while saving) and recalculate the initial rate which will be used next time. For example, if we saved 90 million documents, and another 10 million docs arrived while saving, the rate is 90%, so we know that next time we can collect up to 90% of rt_mem_limit before starting flushing another disk chunk. The rate value is calculated automatically from 33.3% to 95%.indexer -v and --version. Previously you could still see indexer's version, but -v/--version were not supported.MANTICORE_TRACK_RT_ERRORS useful for debugging RT segments corruption./var/lib/manticore/binlog/ except binlog.meta after stopping the previous instance.show threads option format=all. It shows stack of some task info tickets, most useful for profiling needs, so if you are parsing show threads output be aware of the new column.searchd.workers was obsoleted since 3.5.0, now it's deprecated, if you still have it in your configuration file it will trigger a warning on start. Manticore Search will start, but with a warning.PDO::ATTR_EMULATE_PREPARESindextool --check could crashFull support of Manticore Columnar Library. Previously Manticore Columnar Library was supported only for plain indexes. Now it's supported:
INSERT, REPLACE, DELETE, OPTIMIZEALTERindextool --checkAutomatic indexes compaction (Issue #478). Finally, you don't have to call OPTIMIZE manually or via a crontask or other kind of automation. Manticore now does it for you automatically and by default. You can set default compaction threshold via optimize_cutoff global variable.
Chunk snapshots and locks system revamp. These changes may be invisible from outside at first glance, but they improve the behaviour of many things happening in real-time indexes significantly. In a nutshell, previously most Manticore data manipulation operations relied on locks heavily, now we use disk chunk snapshots instead.
Significantly faster bulk INSERT performance into a real-time index. For example on Hetzner's server AX101 with SSD, 128 GB of RAM and AMD's Ryzen™ 9 5950X (16*2 cores) with 3.6.0 you could get 236K docs per second inserted into a table with schema name text, email string, description text, age int, active bit(1) (default rt_mem_limit, batch size 25000, 16 concurrent insert workers, 16 million docs inserted overall). In 4.0.2 the same concurrency/batch/count gives 357K docs per second.
ALTER can add/remove a full-text field (in RT mode). Previously it could only add/remove an attribute.
🔬 Experimental: pseudo-sharding for full-scan queries - allows to parallelize any non-full-text search query. Instead of preparing shards manually you can now just enable new option searchd.pseudo_sharding and expect up to CPU cores lower response time for non-full-text search queries. Note it can easily occupy all existing CPU cores, so if you care not only about latency, but throughput too - use it with caution.
time curl -X POST -d '{"update":{"index":"idx","id":4611686018427387905,"doc":{"mode":0}}}' -H "Content-Type: application/x-ndjson" http://127.0.0.1:6358/json/bulk
real 0m43.783s
user 0m0.008s
sys 0m0.007s
time curl -X POST -d '{"update":{"index":"idx","id":4611686018427387905,"doc":{"mode":0}}}' -H "Content-Type: application/x-ndjson" http://127.0.0.1:6358/json/bulk
real 0m0.006s
user 0m0.004s
sys 0m0.001s
--replay-flags=ignore-trx-errors and --replay-flags=ignore-all-errors so one can still start searchd if the binlog is corruptedcharset_table's default value changes from 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451 to non_cjkOPTIMIZE happens automatically. If you don't need it make sure to set auto_optimize=0 in section searchd in the configuration fileondisk_attrs_default were deprecated, now they are removedtotal in SHOW META, but not total_found which is the actual number of found documents./var/lib/manticore/binlog/ (only binlog.meta should be in the directory)--new-cluster (run tool manticore_new_cluster in Linux).ERROR 1064 (42000): invalid GTID, (null), the donor could become unresponsive while another node was joiningindextool --help doesn't display parameter --rotatecommand_insert, command_replace and others were showing wrong metricscharset_table for a plain index had a wrong default valueSELECT * FROM pq ORDER BY id desc LIMIT 1000 , 100 OPTION max_matches=1100 was not working previouslyMaintenance release before Manticore 4
manticore_new_cluster [--force] useful for restarting a replication cluster via systemdindexer --mergeblend_mode='trim_all'WHERE json.a = 1DEBUG SPLIT as a prerequisite for automatic sharding/rebalancingindextool --dumpheaderreverse_scan is deprecated. Make sure you don't use this option in your queries since 3.6.0 since they will fail otherwisereverse_scan has been deprecatedindexer --all and have not only plain indexes in the configuration file. Without ignore_non_plain=1 you'll get a warning and a respective exit code.indexer --verbose is deprecated as it never added anything to the indexer outputUSR2 is now to be used instead of USR12* No. of cores) instead of a single one. The optimal number of chunks can be controlled by cutoff option.0.SELECT query the number of threads can be limited with OPTION threads=N overriding the global max_threads_per_query./search receives basic support for faceting/grouping by new query node aggs.listen=...:sphinx needs to be explicit set for SphinxSE connections or SphinxAPI clients.killed_documents, killed_rate, disk_mapped_doclists, disk_mapped_cached_doclists, disk_mapped_hitlists and disk_mapped_cached_hitlists.status now outputs Queue\Threads and Tasks\Threads.dist_threads is completely deprecated now, searchd will log a warning if the directive is still used.The official Docker image is now based on Ubuntu 20.04 LTS
Besides the usual manticore package, you can also install Manticore Search by components:
manticore-server-core - provides searchd, manpage, log dir, API and galera module. It will also install manticore-common as the dependency.manticore-server - provides automation scripts for core (init.d, systemd), and manticore_new_cluster wrapper. It will also install manticore-server-core as the dependency.manticore-common - provides config, stopwords, generic docs and skeleton folders (datadir, modules, etc.)manticore-tools - provides auxiliary tools ( indexer, indextool etc.), their manpages and examples. It will also install manticore-common as the dependency.manticore-icudata (RPM) or manticore-icudata-65l (DEB) - provides ICU data file for icu morphology usage.manticore-devel (RPM) or manticore-dev (DEB) - provides dev headers for UDFs.This release took so long, because we were working hard on changing multitasking mode from threads to coroutines. It makes configuration simpler and queries parallelization much more straightforward: Manticore just uses given number of threads (see new setting threads) and the new mode makes sure it's done in the most optimal way.
Changes in highlighting:
highlight({},'field1, field2') or highlight in json queries) now applies limits per-field by default.highlight({}, string_attr) or snippet() now applies limits to the whole document.limits_per_field=0 option (1 by default).0 by default for highlighting via HTTP JSON.The same port can now be used for http, https and binary API (to accept connections from a remote Manticore instance). listen = *:mysql is still required for connections via mysql protocol. Manticore now detects automatically the type of client trying to connect to it except for MySQL (due to restrictions of the protocol).
In RT mode a field can now be text and string attribute at the same time - GitHub issue #331.
In plain mode it's called sql_field_string. Now it's available in RT mode for real-time indexes too. You can use it as shown in the example:
create table t(f string attribute indexed);
insert into t values(0,'abc','abc');
select * from t where match('abc');
+---------------------+------+
| id | f |
+---------------------+------+
| 2810845392541843463 | abc |
+---------------------+------+
1 row in set (0.01 sec)
mysql> select * from t where f='abc';
+---------------------+------+
| id | f |
+---------------------+------+
| 2810845392541843463 | abc |
+---------------------+------+
1 row in set (0.00 sec)
status command.in is now available via HTTP JSON interface.expressions in HTTP JSON.rt_mem_limit on the fly in RT mode, i.e. can do ALTER ... rt_mem_limit=<new value>.chinese, japanese and korean.SHOW THREADS output.CALL PQ in SHOW THREADS.SET [GLOBAL] wait_timeout=NUM implemented ,INSERT INTO PQ VALUES() (i.e. without providing column list) previously expected exactly (query, tags) as the values. It's been changed to (id,query,tags,filters). The id can be set to 0 if you want it to be auto-generated.allow_empty=0 is a new default in highlighting via HTTP JSON interface.CREATE TABLE/ALTER TABLE.ram_chunks_count was renamed to ram_chunk_segments_count in SHOW INDEX STATUS.workers is obsolete. There's only one workers mode now.dist_threads is obsolete. All queries are as much parallel as possible now (limited by threads and jobs_queue_size).max_children is obsolete. Use threads to set the number of threads Manticore will use (set to the # of CPU cores by default).queue_max_length is obsolete. Instead of that in case it's really needed use jobs_queue_size to fine-tune internal jobs queue size (unlimited by default)./json/* endpoints are now available w/o /json/, e.g. /search, /insert, /delete, /pq etc.field meaning "full-text field" was renamed to "text" in describe.
<!-- more -->
3.4.2:
mysql> describe t;
+-------+--------+----------------+
| Field | Type | Properties |
+-------+--------+----------------+
| id | bigint | |
| f | field | indexed stored |
+-------+--------+----------------+
mysql> describe t;
+-------+--------+----------------+
| Field | Type | Properties |
+-------+--------+----------------+
| id | bigint | |
| f | text | indexed stored |
+-------+--------+----------------+
и doesn't map to i in non_cjk charset_table (which is a default) as it affected Russian stemmers and lemmatizers too much.read_timeout. Use network_timeout instead which controls both reading and writing.manticore-bin to manticorecount(*) shows different numbers/json/replace and json/update return id in exponent formhitless_words doesn't work in RT modeALTER RECONFIGURE in rt mode should be disallowedrt_mem_limit gets reset to 128M after searchd restartSHOW CREATE TABLE vs multiple wordform filesSHOW CREATE TABLE doesn't work for PQCREATE TABLE LIKE doesn't work properly for PQCREATE TABLE LIKE infix errorALTER reconfigure corrupts a PQ indexHIGHLIGHT() doesn't higlight in string attributesFACET fails to sort on string attributeCALL PQ returns "Bad JSON objects in strings: 1" when the json is greater than some value.max_xmlpipe2_field = 2M returned warning on 2M field[null] in json attr in centos 7 causes corrupted inserted data/sql HTTP endpoint response is now the same as /json/search responseaccess_plain_attrs, access_blob_attrs, access_doclists, access_hitlistsserver_id for replication setups/search endpointread_buffer, ondisk_attrs, ondisk_attrs_default, mlock are replaced by access_* directivesCmake minimum version is now 3.13. Compiling requires boost and libssl development libraries.
format=sphinxql prints all queries in SQL formatclone_attrs stageshutdown using DEBUG commanddocs_id option for documents called in CALL PQ.SET wait_timeout (for better ProxySQL compatibility)-DUSE_JEMALLOC=1In this release we've changed internal protocol used by masters and agents to speak with each other. In case you run Manticoresearch in a distributed environment with multiple instances make sure your first upgrade agents, then the masters.
multiplier row when multi-query optimization is usedManticore Search is built using cmake and the minimum gcc version required for compiling is 4.7.2.
manticore user./var/lib/manticore/./var/log/manticore/./var/run/manticore/.