ContextQMD
Back to Clickhouse

2025 Changelog

docs/changelogs/v25.3.1.2703-lts.md

26.4.1.1-new70.8 KB
Original Source

2025 Changelog

ClickHouse release v25.3.1.2703-lts (b684e195f96) FIXME as compared to v25.3.1.1-new (217f67ec3ad)

Backward Incompatible Change

  • Disallow truncate database for replicated databases. #76651 (Bharat Nallan).
  • Disable parallel replicas by default when analyzer is disabled regardless compatibility setting. It's still possible to change this behavior by explicitly setting parallel_replicas_only_with_analyzer to false. #77115 (Igor Nikonov).
  • It's no longer possible to use NaN or inf for float values as settings. #77546 (Yarik Briukhovetskyi).

New Feature

  • A new implementation of the Userspace Page Cache, which allows caching data in the in-process memory instead of relying on the OS page cache, which is useful when the data is stored on a remote virtual filesystem without backing with the local filesystem cache. #70509 (Michael Kolupaev).
  • Backported in #77924: Support low cardinality decimal data types, fix #72256. #72833 (zhanglistar).
  • Added concurrent_threads_scheduler server setting that governs how CPU slots are distributed among concurrent queries. Could be set to round_robin (previous behavior) or fair_round_robin to address the issue of unfair CPU distribution between INSERTs and SELECTs. #75949 (Sergei Trifonov).
  • Added function arraySymmetricDifference. It returns all elements from multiple array arguments which do not occur in all arguments. Example: SELECT arraySymmetricDifference([1, 2], [2, 3]) returns [1, 3]. (issue #61673). #76231 (Filipp Abapolov).
  • Add estimatecompressionratio aggregate function- see #70801. #76661 (Tariq Almawash).
  • FilterTransformPassedRows and FilterTransformPassedBytes profile events will show the number of rows and number of bytes filtered during the query execution. #76662 (Onkar Deshpande).
  • Added the keccak256 hash function, commonly used in blockchain implementations, especially in EVM-based systems. #76669 (Arnaud Briche).
  • The functionality adds the ability to define a list of headers that are forwarded from the headers of the client request to the external http authenticator. #77054 (inv2004).
  • Add 3 new functions. icebergTruncate according to specification. https://iceberg.apache.org/spec/#truncate-transform-details, toYearNumSinceEpoch and toMonthNumSinceEpoch. Support truncate transform in partition pruning for Iceberg engine. #77403 (alesapin).
  • Backported in #77975: JSON data type is production-ready. See https://jsonbench.com/. Dynamic and Varaint data types are production ready. #77785 (Alexey Milovidov).

Experimental Feature

  • Added support for dynamic cluster autodiscovery. This extends the existing node autodiscovery feature. ClickHouse can now automatically detect and register new clusters under a common ZooKeeper path using <multicluster_root_path>. #76001 (Anton Ivashkin).
  • Allow automatic cleanup merges of entire partitions after a configurable timeout with a new setting enable_replacing_merge_with_cleanup_for_min_age_to_force_merge. #76440 (Christoph Wurm).
  • Add support for Unity Catalog for DeltaLake tables on top of AWS S3 and local filesystem. #76988 (alesapin).
  • Introduce experimental integration with AWS Glue service catalog for Iceberg tables. #77257 (alesapin).

Performance Improvement

  • Apply preserve_most attribute at some places in code. #67778 (Nikita Taranov).
  • Implement query condition cache to improve query performance using repeated conditions. The range of the portion of data that does not meet the condition is remembered as a temporary index in memory. Subsequent queries will use this index. close #67768 ### Documentation entry for user-facing changes. #69236 (zhongyuankai).
  • Replace Int256 and UInt256 with clang builtin i256 in arithmetic calculation according to tests in #70502. #73658 (李扬).
  • In some cases (e.g. empty array column) data parts can contain empty files. We can skip writing empty blobs to ObjectStorage and only store metadata for such files when the table resides on disk with separated metadata and object storages. #75860 (Alexander Gololobov).
  • Improve min/max performance for Decimal32/Decimal64/DateTime64. #76570 (李扬).
  • Actively evict data from the cache on parts removal. Do not let the cache grow to the maximum size if the amount of data is less. #76641 (Alexey Milovidov).
  • Query compilation (setting compile_expressions) now considers the machine type. This speeds up such queries significantly. #76753 (Robert Schulze).
  • Backported in #77880: Optimize arraySort. #76850 (李扬).
  • Backported in #77961: Disable filesystem_cache_prefer_bigger_buffer_size when the cache is used passively, such as for merges. #77898 (Kseniia Sumarokova).

Improvement

  • Replace table functions with their -Cluster alternatives if parallel replicas are enabled. Fixes #65024. #70659 (Konstantin Bogdanov).
  • Respect column insensitive column matching for fields in tuple columns. Close https://github.com/apache/incubator-gluten/issues/8324. #73780 (李扬).
  • Parameters for the codec Gorilla will now always be saved in the table metadata in .sql file. This closes: #70072. #74814 (Nikita Mikhaylov).
  • Backported in #77949: Enabled SSH protocol back. Fixed some critical vulnerabilities so that it is no longer possible to use custom pager or specify server-logs-file. Disabled the ability to pass client options through the environment variables by default (it is still possible via ssh-server.enable_client_options_passing in config.xml). Supported progress table, query cancellation, completion, profile events progress, stdin and send_logs_level option. This closes: #74340. #74989 (Nikita Mikhaylov).
  • Implemented parsing enhancements (Sequence ID parsing: Added functionality to parse sequence identifiers in manifest files AND Avro metadata parsing: Redesigned the Avro metadata parser to be easily extendable for future enhancements). #75010 (Daniil Ivanik).
  • We have support for gauge/counter metric types. However, they are insufficient for some metrics (e.g., the response times of requests to the keeper), so support for the histogram metric type is needed. The interface closely mirrors the Prometheus client, where you simply call observe(value) to increment the counter in the bucket corresponding to the value. The histogram metrics are exposed via system.histogram_metrics. #75736 (Miсhael Stetsyuk).
  • Remove trace_id from default ORDER BY for system.opentelemetry_span_log. #75907 (Azat Khuzhin).
  • Encryption (XML attribute encrypted_by) can now be applied to any configuration file (config.xml, users.xml, nested configuration files). Previously, it worked only for the top-level config.xml file. #75911 (Mikhail Gorshkov).
  • Convert join to in subquery if output column is tied to the left table, need a uniqueness step at first, so disabled by default until the step is added later. #75942 (Shichao Jin).
  • Improve the system.warnings table and add some dynamic warning messages that can be added, updated or removed. #76029 (Bharat Nallan).
  • This PR makes it impossible to run a query ALTER USER user1 ADD PROFILES a, DROP ALL PROFILES because all DROP operations should come first in the order. #76242 (pufit).
  • Various enhancements for SYNC REPLICA (better error messages, better tests, sanity checks). #76307 (Azat Khuzhin).
  • Use correct fallback when multipart copy to S3 fails during backup with Access Denied. Multi part copy can generate Access Denied error when backup is done between buckets that have different credentials. #76515 (Antonio Andelic).
  • Faster ClickHouse Servers shutdown (get rid of 2.5sec delay). #76550 (Azat Khuzhin).
  • Upgraded librdkafka to version 2.8.0 and improved the shutdown sequence for Kafka tables, reducing delays during table drops and server restarts. The engine=Kafka no longer explicitly leaves the consumer group when a table is dropped. Instead, the consumer remains in the group until it is automatically removed after session_timeout_ms (default: 45 seconds) of inactivity. #76621 (filimonov).
  • Fix validation of s3 request settings. #76658 (Vitaly Baranov).
  • Avoid excess allocation in readbufferfroms3 and other remote reading buffers, reduce their memory consumption in half. #76692 (Sema Checherinda).
  • System tables like server_settings or settings have a default value column which is convenient. only merge_tree_settings and replicated_merge_tree_settings do not have that column enabled. #76942 (Diego Nieto).
  • Added ProfileEvents::QueryPreempted, which has the same logic as CurrentMetrics::QueryPreempted. #77015 (VicoWu).
  • Previously database replicated might print credentials specified in a query to logs. This behaviour is fixed. This closes: #77123. #77133 (Nikita Mikhaylov).
  • Bump zstd from 1.5.5 to 1.5.7 which has pretty good performance improvements. #77137 (Pradeep Chhetri).
  • Allow ALTER TABLE DROP PARTITION for plain_rewritable disk. #77138 (Julia Kartseva).
  • Add the ability to randomly sleep up to 500ms independent of part sizes before merges/mutations execution in case of zero-copy replication. #77165 (Alexey Katsman).
  • Support atomic rename when TRUNCATE is used with INTO OUTFILE. Resolves #70323. #77181 (Onkar Deshpande).
  • Allow to explicitly specify metadata file to read for Iceberg with storage/table function setting iceberg_metadata_file_path . Fixes #47412. #77318 (alesapin).
  • Change reverted. #77399 (Yarik Briukhovetskyi).
  • Backup/restore setting allow_s3_native_copy now supports value three possible values: - False - s3 native copy will not be used; - True (old default) - ClickHouse will try s3 native copy first, if it fails then fallback to the reading+writing approach; - 'auto' (new default) - ClickHouse will compare the source and destination credentials first. If they are same, ClickHouse will try s3 native copy and then may fallback to the reading+writing approach. If they are different, ClickHouse will go directly to the reading+writing approach. #77401 (Vitaly Baranov).
  • Skipping index cache is reverted. #77447 (Nikita Mikhaylov).
  • Reduce memory usage during prefetches of JSON column in Wide parts. #77640 (Pavel Kruglov).
  • Support aws session token and environment credentials usage in delta kernel for DeltaLake table engine. #77661 (Kseniia Sumarokova).

Bug Fix (user-visible misbehavior in an official stable release)

  • Fix stuck while processing pending batch for async distributed INSERT (due to i.e. No such file or directory). #72939 (Azat Khuzhin).
  • Improved datetime conversion during index analysis by enforcing saturating behavior for implicit Date to DateTime conversions. This resolves potential index analysis inaccuracies caused by datetime range limitations. This fixes #73307. It also fixes explicit toDateTime conversion when date_time_overflow_behavior = 'ignore' which is the default value. #73326 (Amos Bird).
  • Fix all sort of bugs due to race between UUID and table names (for instance it will fix the race between RENAME and RESTART REPLICA, in case of concurrent RENAME with SYSTEM RESTART REPLICA you may get end up restarting wrong replica, or/and leaving one of the tables in a Table X is being restarted state). #76308 (Azat Khuzhin).
  • Fix data loss when enable async insert and insert into ... from file ... with unequal block size if the first block size < async_max_size but the second block > async_max_size, the second block will not be inserted. these data is left in squashing. #76343 (Han Fei).
  • Renamed field 'marks' to 'marks_bytes' in system.data_skipping_indices. #76374 (Robert Schulze).
  • Fix dynamic filesystem cache resize handling unexpected errors during eviction. #76466 (Kseniia Sumarokova).
  • Fixed used_flag initialization in parallel hash. It might cause a server crash. #76580 (Nikita Taranov).
  • Fix a logical error when calling defaultProfiles() function inside a projection. #76627 (pufit).
  • Do not request interactive basic auth in the browser in Web UI. Closes #76319. #76637 (Alexey Milovidov).
  • Fix THERE_IS_NO_COLUMN exception when selecting boolean literal from distributed tables. #76656 (Yakov Olkhovskiy).
  • The subpath inside the table directory is chosen in a more profound way. #76681 (Daniil Ivanik).
  • Fix an error Not found column in block after altering a table with a subcolumn in PK. After https://github.com/ClickHouse/ClickHouse/pull/72644, requires https://github.com/ClickHouse/ClickHouse/pull/74403. #76686 (Nikolai Kochetov).
  • Add performance tests for null shortcuit and fix bugs. #76708 (李扬).
  • Flush output write buffers before finalizing them. Fix LOGICAL_ERROR generated during the finalization of some output format, e.g. JSONEachRowWithProgressRowOutputFormat. #76726 (Antonio Andelic).
  • Added support for MongoDB binary UUID (#74452) - Fixed WHERE pushdown to MongoDB when using the table function (#72210) - Changed the MongoDB - ClickHouse type mapping such that MongoDB's binary UUID can only be parsed to ClickHouse's UUID. This should avoid ambiguities and surprises in future. - Fixed OID mapping, preserving backward compatibility. #76762 (Kirill Nikiforov).
  • Fix exception handling in parallel prefixes deserialization of JSON subcolumns. #76809 (Pavel Kruglov).
  • Fix lgamma function behavior for negative integers. #76840 (Ilya Kataev).
  • Fix reverse key analysis for explicitly defined primary keys. Similar to #76654. #76846 (Amos Bird).
  • Fix pretty print of Bool values in JSON format. #76905 (Pavel Kruglov).
  • Fix possible crash because of bad JSON column rollback on error during async inserts. #76908 (Pavel Kruglov).
  • Previously, multi_if may return different types of columns during planning and main execution. This resulted in code producing undefined behavior from the C++ perspective. #76914 (Nikita Taranov).
  • Fixed incorrect serialization of constant nullable keys in MergeTree. This fixes #76939. #76985 (Amos Bird).
  • Fix sorting of BFloat16 values. This closes #75487. This closes #75669. #77000 (Alexey Milovidov).
  • Bug fix json with variant subcolumn by adding check to skip ephemeral subcolumns in part consistency check. #72187. #77034 (Smita Kulkarni).
  • Fix crash in template parsing in Values format in case of types mismatch. #77071 (Pavel Kruglov).
  • Don't allow creating EmbeddedRocksDB table with subcolumn in primary key. Previosly such table could be created but select queries were failing. #77074 (Pavel Kruglov).
  • Fix illegal comparison in distributed queries because pushing down predicates to remote doesn't respect literal types. #77093 (Duc Canh Le).
  • Fix crash during Kafka table creation with exception. #77121 (Pavel Kruglov).
  • Support new JSON and subcolumns in Kafka and RabbitMQ engines. #77122 (Pavel Kruglov).
  • Fix exceptions stack unwinding on MacOS. #77126 (Eduard Karacharov).
  • Fix reading 'null' subcolumn in getSubcolumn function. #77163 (Pavel Kruglov).
  • Fix bloom filter index with Array and not supported functions. #77271 (Pavel Kruglov).
  • We should only check the restriction on the amount of tables during the initial CREATE query. #77274 (Nikolay Degterinsky).
  • SELECT toBFloat16(-0.0) == toBFloat16(0.0) now correctly returns true (from previously false). This makes the behavior consistent with Float32 and Float64. #77290 (Shankar Iyer).
  • Fix posbile incorrect reference to unintialized key_index variable, which may lead to crash in debug builds (this uninitialized reference won't cause issues in release builds because subsequent code are likely to throw errors.) ### documentation entry for user-facing changes. #77305 (wxybear).
  • Reverted. #77307 (Nikolai Kochetov).
  • Fix name for partition with a Bool value. It was broken in https://github.com/ClickHouse/ClickHouse/pull/74533. #77319 (Pavel Kruglov).
  • Fix comparison between tuples with nullable elements inside and strings. As an example, before the change comparison between a Tuple (1, null) and a String '(1,null)' would result in an error. Another example would be a comparison between a Tuple (1, a), where a is a Nullable column, and a String '(1, 2)'. This change addresses these issues. #77323 (Alexey Katsman).
  • Fix crash in ObjectStorageQueueSource. Was intoduced in https://github.com/ClickHouse/ClickHouse/pull/76358. #77325 (Pavel Kruglov).
  • Backported in #77877: Fix a bug when close_session query parameter didn't have any effect leading to named sessions being closed only after session_timeout. #77336 (Alexey Katsman).
  • Fix async_insert with input(). #77340 (Azat Khuzhin).
  • Fix: WITH FILL may fail with NOT_FOUND_COLUMN_IN_BLOCK when sorting column is removed by planer. Similar issue related to inconsistent DAG calculated for INTERPOLATE expression. #77343 (Yakov Olkhovskiy).
  • Reverted. #77390 (Vladimir Cherkasov).
  • Fix several LOGICAL_ERRORs around setting alias of invalid AST nodes. #77445 (Raúl Marín).
  • In filesystem cache impementation fix error processing during file segment write. #77471 (Kseniia Sumarokova).
  • Make DatabaseIceberg use correct metadata file provided by catalog. Closes #75187. #77486 (Kseniia Sumarokova).
  • Revert 'Avoid toAST() in execution of scalar subqueries'. #77584 (Raúl Marín).
  • The query cache now assumes that UDFs are non-deterministic. Accordingly, results of queries with UDFs are no longer cached. Previously, users were able to define non-deterministic UDFs whose result would erronously be cached (issue #77553). #77633 (Jimmy Aguilar Mena).
  • Fix system.filesystem_cache_log working only under setting enable_filesystem_cache_log. #77650 (Kseniia Sumarokova).
  • Fix a logical error when calling defaultRoles() function inside a projection. Follow-up for #76627. #77667 (pufit).
  • Second arguments of type Nullable for function arrayResize are now disallowed. Previously, anything from errors to wrong results could happen with Nullable as second argument. (issue #48398). #77724 (Manish Gill).
  • Backported in #77821: Regularly check if merges and mutations were cancelled even in case when the operation doesn't produce any blocks to write. #77766 (János Benjamin Antal).

Build/Testing/Packaging Improvement

NO CL CATEGORY

NO CL ENTRY

NOT FOR CHANGELOG / INSIGNIFICANT