docs/changelogs/v24.8.1.2684-lts.md
clickhouse-client and clickhouse-local now default to multi-query mode (instead single-query mode). As an example, clickhouse-client -q "SELECT 1; SELECT 2" now works, whereas users previously had to add --multiquery (or -n). The --multiquery/-n switch became obsolete. INSERT queries in multi-query statements are treated specially based on their FORMAT clause: If the FORMAT is VALUES (the most common case), the end of the INSERT statement is represented by a trailing semicolon ; at the end of the query. For all other FORMATs (e.g. CSV or JSONEachRow), the end of the INSERT statement is represented by two newlines \n\n at the end of the query. #63898 (FFish).LowCardinality data types by appending WithDictionary to the name of the data type. It was an initial working implementation, and it was never documented or exposed to the public. Now, it is deprecated. If you have used this syntax, you have to ALTER your tables and rename the data types to LowCardinality. #66842 (Alexey Milovidov).Buffer used with distributed destination table. It's a backward incompatible change: queries using Buffer with a distributed destination table may stop working if the table appears more than once in the query (e.g., in a self-join). #67015 (vdimir).text_log is enabled by default. This is fully compatible with previous versions, but you may notice subtly increased disk usage on the local disk (this system table takes a tiny amount of disk space). #67428 (Alexey Milovidov).arrayWithConstant can be slow if asked to generate very large arrays. In the new version, it is limited to 1 GB per array. This closes #32754. #67741 (Alexey Milovidov).CREATE TABLE tbl ENGINE=TimeSeries - or with specifying engines of its internal tables:. #64183 (Vitaly Baranov).LEFT/RIGHT SEMI/ANTI/ANY JOIN) with inequal conditions which involve columns from both left and right table. e.g. t1.y < t2.y (see setting allow_experimental_join_condition). #64281 (lgbo)._etag virtual column for S3 table engine. Fixes #65312. #65386 (skyoct).File, URL, S3, AzureBlobStorage, HDFS). Hive-style partitioning organizes data into partitioned sub-directories, making it efficient to query and manage large datasets. Currently, it only creates virtual columns with the appropriate name and data. The follow-up PR will introduce the appropriate data filtering (performance speedup). #65997 (Yarik Briukhovetskyi).restore_replace_external_engines_to_null and restore_replace_external_table_functions_to_null to replace external engines and table_engines to Null engine that can be useful for testing. It should work for RESTORE and explicit table creation. #66536 (Ilya Yatsishin).fuzzQuery. This function allows the modification of a given query string with random variations. Example: SELECT query FROM fuzzQuery('SELECT 1') LIMIT 5;. #67655 (pufit).DROP DETACHED PARTITION ALL to drop all detached partitions. #67885 (Duc Canh Le).SELECT 1 SETTINGS use_query_cache = 1, query_cache_tag = 'abc' and SELECT 1 SETTINGS use_query_cache = 1, query_cache_tag = 'def' now create different query cache entries. #68235 (sakulali).plain_rewritable disk directory metadata in __meta layout, separately from the merge tree data in the object storage. Move the plain_rewritable disk to a flat directory structure. #65751 (Julia Kartseva).compile_expressions (JIT compiler for fragments of ordinary expressions) by default. This closes #51264 and #56386. #66486 (Alexey Milovidov).optimize_functions_to_subcolumns is enabled by default. #68053 (Anton Popov).max_keep_alive_requests. For keep-alive HTTP connections to the server it works in tandem with keep_alive_timeout - if idle timeout not expired but there already more than max_keep_alive_requests requests done through the given connection - it will be closed by the server. #61793 (Nikita Taranov).count_min (count-min sketches) which provide selectivity estimations for equality predicates like col = 'val'. Supported data types are string, date, datatime and numeric types. #65521 (JackyWoo).Atomic database by default in clickhouse-local. Address items 1 and 5 from #50647. Closes #44817. #65860 (Alexey Milovidov).rows_before_aggregation_at_least statistic to the query response when rows_before_aggregation is enabled. This statistic represents the number of rows read before aggregation. In the context of a distributed query, when using the group by or max aggregation function without a limit, rows_before_aggregation_at_least can reflect the number of rows hit by the query. #66084 (morning-color).use_same_password_for_base_backup settings for BACKUP and RESTORE queries, allowing to create and restore incremental backups to/from password protected archives. #66214 (Samuele).onCancel() method. #66279 (Sema Checherinda).allow_experimental_analyzer is renamed to enable_analyzer. The old name is preserved in a form of an alias. #66438 (Nikita Mikhaylov).MergeTree setting deduplicate_merge_projection_mode to control the projections during merges (for specific engines) and OPTIMIZE DEDUPLICATE query. Supported options: throw (throw an exception in case the projection is not fully supported for *MergeTree engine), drop (remove projection during merge if it can't be merged itself consistently) and rebuild (rebuild projection from scratch, which is a heavy operation). #66672 (jsc0218).BAD_ARGUMENTS and ILLEGAL_TYPE_OF_ARGUMENT by more accurate error codes when appropriate. #66851 (Yohann Jardin).mapFromArrays now accepts Map(K, V) as first argument, for example: SELECT mapFromArrays(map('a', 4, 'b', 4), ['aa', 'bb']) now works and returns {('a',4):'aa',('b',4):'bb'}. Also, if the 1st argument is an Array, it can now also be of type Array(Nullable(T)) or Array(LowCardinality(Nullable(T))) as long as the actual array values are not NULL. #67103 (李扬).input_format_orc_read_use_writer_time_zone to input_format_orc_reader_timezone and allow the user to set the reader timezone. #67175 (kevinyhzou).DROP DATABASE query, increased the default value for database_catalog_drop_table_concurrency to 16. #67228 (Nikita Mikhaylov).denseRank() and percentRank() have been created. These new functions can be called the exact same as the original dense_rank() and percent_rank() functions. Both snake case and camelCase syntaxes remain usable. A new test for each of the functions has been added as well. This closes #67042 . #67334 (Peter Nguyen).formatDateTime and formatDateTimeInJodaSyntax now treat their format parameter as optional. If it is not specified, format strings %Y-%m-%d %H:%i:%s and yyyy-MM-dd HH:mm:ss are assumed. Example: SELECT parseDateTime('2021-01-04 23:12:34') now returns DateTime value 2021-01-04 23:12:34 (previously, this threw an exception). #67399 (Robert Schulze).date_time_input_format='best_effort' to avoid inferring date times from strings in corner cases. #68382 (Kruglov Pavel).Not-ready Set in some system tables when filtering using subqueries. #66018 (Michael Kolupaev).ALTER ADD COLUMN query. #66243 (Anton Popov).PostgreSQL). #66282 (vdimir).... JOIN t2 ON (x = y) AS e ORDER BY x should be formatted as ... JOIN t2 ON ((x = y) AS e) ORDER BY x. #66312 (vdimir).s3_plain_rewritable disk. #66881 (Vitaly Baranov).numbers could wrongly throw an exception when the max_rows_to_read limit was set. This closes #66992. #66996 (Alexey Milovidov).clickhouse-local. #67133 (Vitaly Baranov).Cannot convert column because it is non constant in source stream but must be constant in result. for a query that reads from the Merge table over the Distriburted table with one shard. #67146 (Nikolai Kochetov).ORDER BY all with disabled enable_order_by_all and parallel replicas (distributed queries as well). #67153 (Igor Nikonov).Conversion from AggregateFunction(name, Type) to AggregateFunction(name, Nullable(Type)) is not supported. The bug was caused by the optimize_rewrite_aggregate_function_with_if optimization. Fixes #67112. #67229 (Nikolai Kochetov).uniq and uniqTheta with tuple() argument. Closes #67303. #67306 (flynn).Aggregator. #67385 (Antonio Andelic).bloom_filter index breaking queries with mildly weird conditions like (k=2)=(k=2) or has([1,2,3], k). #67423 (Michael Kolupaev).:: if it's not an archive. #67433 (Antonio Andelic).ReplaceTableNodeToDummyVisitor. #67522 (Dmitry Novik).Logical error: Expected the argument №N of type T to have X rows, but it has 0. The error could happen in a remote query with constant expression in GROUP BY (with a new analyzer). #67536 (Nikolai Kochetov).NULL inside the tuple in the JOIN ON section returned incorrect results. #67538 (vdimir).toStartOfWeek which returned the wrong result with a small DateTime64 value. #67558 (Yarik Briukhovetskyi).Logical error: 'file_offset_of_buffer_end <= read_until_position' in filesystem cache. Closes #57508. #67623 (Kseniia Sumarokova).convertFieldToString() and added datatype specific serialization code. Parameterized view substitution was broken for multiple datatypes when parameter value was a function or expression returning datatype instance. #67654 (Shankar).percent_rank. percent_rank's default frame type is changed to range unbounded preceding and unbounded following. IWindowFunction's default window frame is considered and now window functions without window frame definition in sql can be put into different WindowTransfomers properly. #67661 (lgbo).use_variant_as_common_type in function if with Tuples and Maps. #67687 (Kruglov Pavel).TimerDescriptor::drain. This closes #37686. #67702 (Alexey Milovidov).RESTORE ON CLUSTER command. #67720 (Vitaly Baranov).JSONMergePatch function. Renamed this function from jsonMergePatch to JSONMergePatch because the previous name was wrong. The previous name is still kept for compatibility. Improved diagnostic of errors in the function. This closes #67304. #67756 (Alexey Milovidov).SELECT count() FROM t WHERE cast(c = 1 or c = 9999 AS Bool) SETTINGS use_skip_indexes=1 with bloom filter indexes on c now work correctly. #67781 (jsc0218).count() result when there is non-deterministic function in predicate. #67922 (János Benjamin Antal).checksums.txt. #68003 (alesapin).OFFSET. Fixes #67906. #68099 (Graham Campbell).Block structure mismatch in AggregatingStep stream: different types for aggregate projection optimization. #68107 (Nikolai Kochetov).SYSTEM SYNC REPLICA. #68326 (Duc Canh Le).test_storage_s3 tests: increased s3_max_single_read_retries for read from "unstable" s3 source and allowed all tests to run multiple times in a row. #66896 (Ilya Yatsishin).pytest-repeat library to run test case multiple times for the same environment. It is important to cleanup tables and other entities in the end of a test case to pass. Repeat works much faster than several pytest runs as it starts necessary containers only once. #66986 (Ilya Yatsishin).compile_expressions by default."'. #67299 (Alexey Milovidov).Atomic database by default in clickhouse-local"'. #68023 (Alexey Milovidov).02814_currentDatabase_for_table_functions. #66111 (Nikita Mikhaylov).ClientApplicationBase. #66549 (Nikita Mikhaylov).01042_system_reload_dictionary_reloads_completely. #66811 (Alexey Milovidov).trace_profile_events. #66821 (Alexey Milovidov).share_big_sets, CC @davenger. #66908 (Alexey Milovidov).test_broken_projections/test.py::test_broken_ignored_replicated. #66915 (Andrey Zvonov).01923_network_receive_time_metric_insert. #66924 (János Benjamin Antal).01454_storagememory_data_race_challenge. #67003 (Antonio Andelic).very_long_arrays. #67009 (Alexey Milovidov).test_backup_restore_on_cluster/test_concurrency.py & test_manipulate_statistics/test.py. #67027 (Nikita Fomichev).optimize_functions_to_subcolumns. #67046 (Anton Popov).** to hdfs docs, add test for ** in hdfs. #67064 (Andrey Zvonov).2024.07.24 13:28:45.517777 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> executeQuery: (from 172.16.11.1:55890) OPTIMIZE TABLE replicated_mt FINAL (stage: Complete) 2024.07.24 13:28:45.525945 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Trace> default.replicated_mt (ReplicatedMergeTreeQueue): Waiting for 4 entries to be processed: queue-0000000004, queue-0000000002, queue-0000000001, queue-0000000000 2024.07.24 13:29:15.528024 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> default.replicated_mt (6581a6fb-8458-466d-8350-89951eb1ac8e) (MergerMutator): Selected 3 parts from all_0_0_0 to all_2_2_0 2024.07.24 13:29:15.530736 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Trace> default.replicated_mt (6581a6fb-8458-466d-8350-89951eb1ac8e): Created log entry /clickhouse/tables/replicated_mt/log/log-0000000004 for merge all_0_2_1 2024.07.24 13:29:15.530873 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> default.replicated_mt (6581a6fb-8458-466d-8350-89951eb1ac8e): Waiting for node1 to process log entry 2024.07.24 13:29:15.530919 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> default.replicated_mt (6581a6fb-8458-466d-8350-89951eb1ac8e): Waiting for node1 to pull log-0000000004 to queue 2024.07.24 13:29:15.534286 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> default.replicated_mt (6581a6fb-8458-466d-8350-89951eb1ac8e): Looking for node corresponding to log-0000000004 in node1 queue 2024.07.24 13:29:15.534793 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> default.replicated_mt (6581a6fb-8458-466d-8350-89951eb1ac8e): Waiting for queue-0000000005 to disappear from node1 queue 2024.07.24 13:29:15.585533 [ 10 ] {08745bf9-4bc1-4946-b9a8-c03d82ec55dc} <Debug> TCPHandler: Processed in 30.067804125 sec.. #67067 (alesapin).test_seekable_formats_url and test_seekable_formats S3 storage tests. #67070 (János Benjamin Antal).00673_subquery_prepared_set_performance. #67141 (Alexey Milovidov).02481_async_insert_race_long flakiness. #67148 (Julia Kartseva).00705_drop_create_merge_tree. #67170 (Alexey Milovidov).test_pkill_query_log (tsan). #67223 (Sergei Trifonov).test_broken_projections_in_backups_1. #67231 (Vitaly Baranov).02310_clickhouse_local_INSERT_progress_profile_events. #67264 (Alexey Milovidov).02982_aggregation_states_destruction. #67266 (Alexey Milovidov).03201_variant_null_map_subcolumn. #67276 (Alexey Milovidov).01651_lc_insert_tiny_log. #67279 (Alexey Milovidov).02490_benchmark_max_consecutive_errors. #67281 (Alexey Milovidov).02833_concurrrent_sessions. #67282 (Alexey Milovidov).01600_parts_states_metrics_long. #67284 (Alexey Milovidov).02231_buffer_aggregate_states_leak. #67285 (Alexey Milovidov).TimerDescriptor. #67287 (Alexey Milovidov).02911_backup_restore_keeper_map. #67290 (Alexey Milovidov).compile_expressions. #67300 (Alexey Milovidov).02050_client_profile_events. #67309 (Alexey Milovidov).00940_max_parts_in_total. #67313 (Alexey Milovidov).test_backup_restore_on_cluster/test_disallow_concurrency. #67336 (Vitaly Baranov).test_storage_kerberized_kafka. #67349 (Andrey Zvonov).01036_no_superfluous_dict_reload_on_create_database. #67390 (Alexey Milovidov).SelectedPartsTotal and SelectedMarksTotal as new ProfileEvents. #67393 (Jordi Villar).test_storage_s3_queue/test.py::test_shards_distributed. #67394 (János Benjamin Antal).--multiquery parameter from tests. #67435 (Robert Schulze).StorageURLSource. #67455 (Antonio Andelic).01923_network_receive_time_metric_insert.sh. #67492 (Julia Kartseva).test_backup_restore_on_cluster/test.py::test_mutation. #67494 (Vitaly Baranov).test_storage_azure_blob_storage/test.py is flaky. #67512 (Daniil Ivanik).stop_logs_replication step. Add a check for timeout here. #67560 (Nikolay Degterinsky).test_system_kafka_consumers_rebalance. #67566 (János Benjamin Antal).no-parallel tags from tests. #67610 (Raúl Marín).02481_async_insert_race_long.sh flakiness fixes. #67650 (Julia Kartseva).test_replicated_table_attach. #67658 (Antonio Andelic).no-parallel tags from tests (Part 2). #67673 (Raúl Marín).00002_log_and_exception_messages_formatting. #67723 (Alexey Milovidov).02789_reading_from_s3_with_connection_pool. #67726 (Alexey Milovidov).02473_multistep_prewhere* 00411_long_accurate_number_comparison*. #67746 (Nikita Fomichev).no-flaky-check tag. #67755 (Alexey Milovidov).02833_concurrent_sessions, Fix test 02835_drop_user_during_session. #67779 (Alexey Milovidov).02231_bloom_filter_sizing. #67784 (Alexey Milovidov).test_dictionaries_update_and_reload::test_reload_after_fail_by_timer. #67793 (Nikita Fomichev).03032_redundant_equals. #67822 (Alexey Milovidov).ALTER MODIFY SQL SECURITY on non-view tables. #67953 (pufit).test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed. #67959 (Julia Kartseva).--multiquery parameter (follow-up to #63898), pt. III. #67964 (Robert Schulze).02845_threads_count_in_distributed_queries. #68011 (Alexey Milovidov).00284_external_aggregation.sql. #68037 (Robert Schulze).trace_profile_events in clickhouse-test. #68058 (Alexey Milovidov).Buffer's max time to flush. This PR doubles the timeout and allows to skip the check in case of significant latency. #68072 (pufit).02675_profile_events_from_query_log_and_client. #68097 (Antonio Andelic).WithRetries. #68106 (Antonio Andelic).00900_long_parquet_load. #68130 (Alexey Milovidov).test_cluster_all_replicas. #68178 (Alexey Milovidov).01172_transaction_counters. #68182 (Alexey Milovidov).