What's new in the v2.23 preview release series

Release announcements

• Enhanced Cron Job Resilience With pg_cron in YugabyteDB

v2.23.1.0 - November 12, 2024 {#v2.23.1.0}

Build: 2.23.1.0-b220

Downloads

Docker:

docker pull yugabytedb/yugabyte:2.23.1.0-b220

Change log

Improvements

YSQL

• Changes the CatalogCacheMisses metric name to showcase separate cache misses for each catalog cache, assisting users in easily preloading necessary catalog tables. {{<issue 22843>}}
• Enables in-place updates for non-key columns in indexes, reducing number of flushes. {{<issue 20908>}}
• Introduces the yb_make_next_ddl_statement_nonincrementing configuration parameter to reduce catalog version increments during many consecutive DDLs, lowering memory and latency spikes. {{<issue 23786>}}
• Updates advisory lock error message with a workaround hint instead of not yet implemented. {{<issue 23881>}}
• Allows certain ALTER VIEW commands in the parser without requiring additional changes. {{<issue 23117>}}
• Includes a modification to ysql_dump and Postgres that ensures all pg_class and pg_type OID preservation across a backup/restore when passing the include-yb-metadata flag. {{<issue 23304>}}
• Streamlines sys table prefetching by automatically resetting catalog read time. {{<issue 23502>}}
• Simplifies several ybc_pggate functions for better value return and removes unused functions. {{<issue 23513>}}
• Revamps the restoration of old backups by only applying the check for pg_class OID if the new YSQL configuration parameter, yb_ignore_heap_pg_class_oids, is enabled. {{<issue 23304>}}
• Unmodified columns on main table will continue to be written out when update optimizations are enabled, removing dependency on the feature controlled by ysql_skip_row_lock_for_update. {{<issue 18822>}}
• Adds table-level Prometheus metrics for catcache misses at node-level, viewable at :13000/prometheus-metrics endpoint. {{<issue 23706>}}
• Auto analyze service now tracks mutation counts from DDL changes and eliminates redundant analysis of indexes. {{<issue 22104>}},{{<issue 23506>}}
• Adds serialization/deserialization and equality checking mechanism for update optimization metadata. {{<issue 18822>}}
• Allows adding a primary key using a unique index by performing a table rewrite. {{<issue 23118>}}
• Enables the batch execution of explicit row lock requests in Pg, improving the handling and performance of such requests. {{<issue 22519>}}
• Allows validation of foreign-key and check constraints with ALTER TABLE ...VALIDATE CONSTRAINT. {{<issue 3946>}}
• Added yb_make_next_ddl_statement_nonincrementing to YbDbAdminVariables for yb_db_admin role. {{<issue 23786>}}
• Allows preloading of foreign key lists in relcache, avoiding on-demand master fetches, controlled by yb_enable_fkey_catcache. {{<issue 23686>}}
• Refines the PatchStatus function by storing relation id directly in the relations vector, avoiding the need to call the PgSession::LoadTable method. {{<issue 24040>}}

YCQL

• Updates YCQL tables system_schema.functions and system_schema.aggregates to align with Cassandra, aiding Cortex migration. {{<issue 3893>}}

DocDB

• Reduces risk of WAL over-garbage collection in xCluster and avoids unnecessary flushes by modifying the WAL retention policy. {{<issue 22862>}}
• Enables reduction of duplicate code and custom flag filtering through GetFlagInfos relocation. {{<issue 23632>}}
• Accelerates xCluster setup for tables by reducing setup time, storing all intermediate state in-memory, and limiting writes to sys_catalog. {{<issue 23286>}}
• Allows adding new CatalogEntries without modifying the sys-catalog-tool through using helper functions. {{<issue 23235>}}
• Enables reading stored vector index data from DocDB. {{<issue 23460>}}
• Enables scanning of the sequences_data table for xCluster replication, aiding in the bootstrapping process by integrating current sequence information into the WAL from the source universe. {{<issue 23493>}}
• Introduces DEFINE_NEW_INSTALL_VALUE macro to enable controlled rollout of new features in fresh clusters. {{<issue 23505>}}
• Enables taking into account of changed data during a vector index data update. {{<issue 23524>}}
• Adds utilities for HNSW vector index implementation, benchmarking, and file reading. {{<issue 23376>}}
• Removes the CallVisitor template function to simplify the sys-catalog-tool code. {{<issue 23548>}}
• Allows each sequence to have an update in the WAL for seamless xCluster replication. {{<issue 23493>}}
• Introduces a command-line tool for building, validating, and benchmarking HNSW indexes with Usearch implementation. {{<issue 23556>}}
• Offers support for different vector coordinate types including vector-related templates. {{<issue 23613>}}
• Imports hnswlib header-only library for enhancements. {{<issue 23609>}}
• Introduces a new gflag max_disk_throughput_mbps for automated control of write rejections when disk is full, replacing reject_writes_min_disk_space_aggressive_check_mb. {{<issue 23373>}}
• Allows easier debugging with additional logging and updated log prefixes. {{<issue 23702>}}
• Ensures that ddl_queue and replicated_ddls do not colocate on the same tablet in colocated databases. {{<issue 23728>}}
• Integrates hnswlib into the vector indexing framework, adding wrapper and allowing different distance calculations. {{<issue 23752>}}
• Eliminates the capability to rename xCluster replication groups, reducing potential confusion. {{<issue 23778>}}
• Enables sequences replication feature in source universe by creating sequences_data table and incorporating an outgoing stream. {{<issue 23917>}}
• Enables shared memory by default for executing read and write pg client queries to enhance performance. {{<issue 23820>}}
• Streamlines automatic syncing of inline third-party dependencies and breaks down thirdparty_tool into separate modules. {{<issue 23846>}}
• Enables automatic application of DDL changes in xCluster using automatic_ddl_mode proto field. {{<issue 23860>}}
• Enables display of cluster_uuid in tserver /varz endpoint for better test and call_home tracking. {{<issue 23864>}}
• Resets the cdc_wal_retention_time_secs value on table removal from xCluster replication and applies updates dynamically, eliminating need to drop and recreate replication. {{<issue 20769>}}
• Ensures correct functioning of search_path by skipping its value's enquote during transaction setup. {{<issue 23440>}}
• Enhances sequence generation consistency by making backend process stick to a logical connection when initializing a sequence variable with nextval function. {{<issue 23601>}}
• Ensures DDL operations success with a sleep interval after database catalog version updates when Connection Manager is active. {{<issue 23756>}}
• Updates the callhome URL to use HTTPS for enhanced security. {{<issue 23947>}}
• Includes PID in yb_active_session_history for easy correlation with actual processes in YSQL, YCQL, and background activities. {{<issue 23070>}}
• Allows reduction of thread stack size to avoid Linux hugepage backing. {{<issue 23927>}}
• Allows explicit addition of gFlags to gflag_allowlist.txt for secure callhome data collection, plus version_info in tserver data. {{<issue 24103>}}
• Adds max_follower_heartbeat_delay metric to alert users when the master follower heartbeat delay is too high. {{<issue 21178>}}
• Removes the need to set set_cdc_min_replicated_index during xCluster bootstrap for improved log retention. {{<issue 24105>}}
• Ensures the user interface accurately displays disk size, even when multiple data directories are used. {{<issue 23810>}}

CDC

• Allows dynamic table addition alongside table removal from CDC streams without needing to disable or restart. {{<issue 23581>}}
• Adds DYNAMIC_TABLES_DISABLED yb-admin option to disable dynamic table addition when creating new streams. {{<issue 23482>}}
• Enables default non-eligible table cleanup in CDC streams, replaces the flag enable_cleanup_of_non_eligible_tables_from_cdcsdk_stream with cdcsdk_enable_dynamic_table_addition_with_table_cleanup, and removes flag protection for table identification. {{<issue 23806>}}
• Syncs table removal from a CDC stream to a more performant async process, allowing both manual and automatic removal via the RemoveUserTableFromCDCSDKStream and RemoveTablesFromCDCSDKStream RPCs respectively. {{<issue 23700>}}
• Disables wal_level checks for logical replication in YSQL as PG WAL is not used. {{<issue 23661>}}
• Changes the default wal_level to logical to ensure compatibility with logical replication clients. {{<issue 23661>}}
• Enables asynchronous removal of user tables from CDC streams by adjusting how background threads process and persist stream metadata. {{<issue 23700>}}
• Enables tablet splitting on tables under CDCSDK stream by default using the GFLAG enable_tablet_split_of_cdcsdk_streamed_tables. {{<issue 24190>}}
• Adds a tag for the slot name attribute in the CDC metrics. {{<issue 24307>}}

yugabyted

• Handles OBJECT_NOT_FOUND errors during SELECT/UPDATE/DELETE after TRUNCATE when Connection Manager is enabled. {{<issue 23668>}}
• Enables bitmap scans and removes size-based fetching in the pg parity feature by updating GFlags. {{<issue 23777>}}
• Updates to the pg parity testcase now remove size-based fetching and include enabling bitmap scans. {{<issue 23777>}}
• Changes the flag name from enable_pg_parity_tech_preview to enable_pg_parity_early_access in all branches. {{<issue 23896>}}
• Stops sending gflags details in the callhome diagnostics to eliminate redundant data. {{<issue 24029>}}

Bug fixes

YSQL

• Removes unnecessary DCHECK for smoother rolling upgrades from release 2.20, enhancing ysql_enable_db_catalog_version_mode gflag functionality. {{<issue 23462>}}
• Fixes an issue related to regular expression pushdown causing inconsistency and crashes by introducing thread local cache. {{<issue 22989>}}
• Resets catalog read time after table prefetching to prevent outdated readings for on-demand loading from a master. {{<issue 23421>}}
• Corrects IndexScan's indextlist to improve accuracy of variable references in subquery scenarios. {{<issue 22533>}}
• Prevents individually moving a colocated table to a different tablespace, forcing users to move all tables collectively. {{<issue 23314>}}
• Minimizes read restart errors during ANALYZE operation by using a fresh read time for each table, finding an optimal balance between read oldness and transactionality. {{<issue 22135>}}
• Reduces the memory consumption of the fk_reference_cache_ in large transactions, thereby ensuring minimal use of RAM. {{<issue 23734>}}
• Prevents retry of schema mismatch errors in batched execution mode to maintain transaction atomicity. {{<issue 23843>}}
• Revamped Storage Read Requests metric now better reflects true RPC count in PrecastRequestSender usage. {{<issue 23396>}},{{<issue 18785>}}
• Resolves INSERT ON CONFLICT bugs in temporary tables to ensure consistent outputs. {{<issue 1999>}}
• Fixes a TupleDesc reference leak warning in nested INSERT ON CONFLICT. {{<issue 23429>}}
• Restores unintentional grammar changes to YSQL ALTER SCHEMA command for bug fix. {{<issue 23457>}}
• Fixes repeated calls in DeleteTableInternal, preventing YbAdminSnapshotScheduleTest.SysCatalogRetention failure on asan builds. {{<issue 23459>}}
• Resolves potential memory corruption by mutating pushdown expressions outside the per tuple context. {{<issue 23461>}}
• Eliminates the publishing of empty metrics from the statementType enum on the YSQL Metrics endpoint. {{<issue 23557>}}
• Simplifies the process of identifying identical data by using storage equality, not semantic equality, during UPDATE queries, handling problems that arise when data types lack equality operators or when user-defined data types exhibit unusual semantic equality. {{<issue 23490>}}
• Addresses a bug in YSQL that caused crashes while costing an index scan on a hash index with included columns using CBO. {{<issue 22772>}}
• Adds more logs to aid in debugging a transaction status mismatch issue. {{<issue 23669>}}
• Fixes the hanging issue when creating a colocated table by correcting the unconditional reading of the relkind column. {{<issue 23708>}}
• Adds log_dist option to auto_explain docs for YSQL. {{<issue 23814>}}
• Addresses the issue of invalid Attnum with Bitmap Scan +subplans +nested loops previously encountered in Index Scans. {{<issue 23596>}}
• Adds table name to table-level metrics on /metrics endpoint for better identification. {{<issue 23707>}}
• Fixes a bug that caused incorrect deducing of commit/abort status for transactions which do not change the table's schema. {{<issue 23669>}}
• Adds HELP and TYPE metadata to YSQL Prometheus metrics endpoint for better metrics accessibility. {{<issue 23578>}}
• Corrects an issue causing heap-buffer-overflow errors during ysqlconnmgr ASAN tests. {{<issue 23913>}}
• Fixes a crash caused by using the pg_hint_plan with the hint_table enabled. {{<issue 23547>}}
• Updates the copyright string from YugaByteDB to the correct version YugabyteDB, ensuring error-free linter runs. {{<issue 23940>}}
• Allows for a modified calculation of net connections in YSQL connection manager with the warmup random mode. {{<issue 23588>}}
• Prevents background parallel workers from executing check/assign hooks of YSQL configuration parameters, eliminating potential crashes. {{<issue 23787>}}
• Modifies backup/restore process to skip column name checks for indexes, allowing for successful restoration even with renamed columns. {{<issue 24207>}}
• Fixes crashes caused by using a RowComparisonExpression on a reordered primary key index. {{<issue 23824>}}
• Improves cache re-invalidation for ALTER TABLE commands to avoid schema version mismatch errors within the same session. {{<issue 23882>}}
• Streamlines ysql_dump process by eliminating faulty statements for colocated table unique constraint, enhancing database reliability. {{<issue 24057>}}
• Introduces batching for the INSERT ON CONFLICT command to enhance performance by minimizing alternating read and write operations. {{<issue 24179>}}
• Fixes the flaw in the current DDL atomicity workflow where only the first table's schema was compared, even if it doesn't change after a schema version increment. {{<issue 23988>}}
• Alters encoding setup order to ensure UTF8 use instead of SQL_ASCII and adds pg_collation to preloaded tables to prevent specific errors. {{<issue 24149>}}
• Corrects a typo to prevent incorrect loading of pg_collation catalog caches, eliminating associated performance issues. {{<issue 24149>}}

YCQL

• Eliminates unexpected error during shell type definition update across multiple sessions. {{<issue 24217>}}

DocDB

• Prevents tablet bootstrap from getting stuck when replaying a truncate operation. {{<issue 23243>}}
• Limits the response size of ListSnapshotSchedules RPC by excluding unnecessary per-object details to prevent overflow issues. {{<issue 23518>}}
• Reduces the risk of deadlocks when making active read/write requests to a table right after a tablet-split. {{<issue 23747>}}
• Simplifies handling of large RPC responses by catching errors earlier and appropriately using narrow_cast. {{<issue 22301>}}
• Fixes errors from executing rpc callback twice due to lack of return after detecting disabled wait-queue or deadlock features. {{<issue 23808>}}
• Switches write query duration from CoarseMonoClock to MonoClock for precise nanosecond granularity. {{<issue 20335>}}
• Sets the close_timestamp_micros field for partially copied log segments to prevent time-based policy violation. {{<issue 23335>}}
• Eliminates persistent logging issue after dropping a colocated table by removing unnecessary logs. {{<issue 22037>}}
• Corrects leader balancing for geopartitioned tables to prevent early exits and ensure accurate global leader sorting. {{<issue 23886>}}
• Adds database_oid to the TSLocalLockManager locking contract to distinguish objects from different databases. {{<issue 23454>}}
• Adds debugging information for retryable requests rejected by followers despite acceptance by leaders. {{<issue 23523>}}
• Corrects typo and includes minimum allowed value in the validation message for xcluster_checkpoint_max_staleness_secs flag. {{<issue 23576>}}
• Prevents tablet partition collision during ImportSnapshot by not cloning DELETED or REPLACED tablets. {{<issue 23552>}}
• Allows pg_cron to correctly handle SIGTERM by using quickdie instead of die or pg_cron_sigterm. {{<issue 23655>}}
• Allows the retry of checkpoint if leader not ready or service unavailable errors occur, and prevents client retries on a failed checkpoint by storing an InternalError status. {{<issue 23719>}}
• Restores transaction_aware field in master_backup.proto for compatibility with older DB versions. {{<issue 23739>}}
• Enhances tablet limit flags by adding more details to the main flag and adjusting the help strings. {{<issue 23695>}}
• Adds filtering for bootstrap intent iterators based on min_replay_txn_start_ht to prevent unnecessary SST files buildup during bootstrap. {{<issue 23890>}}
• Fixes a compile error caused by a mismatch in the declaration of protobuf_message_total_bytes_limit. {{<issue 23828>}}
• Reduces the potential for core dump errors during Point In Time Recovery (PITR) by utilizing shared_ptr in callbacks. {{<issue 23399>}}
• Renames the yql_endpoint_tserver_uuid field to top_level_node_id in /rpcz to maintain consistency. {{<issue 23835>}}
• Ensures correct and consistent updating of grptablespace column in pg_yb_tablegroup after ALTER TABLE command. {{<issue 23336>}}
• Ensures the CREATE TABLE ...TABLESPACE pg_default command in colocated databases aligns with PostgreSQL semantics. {{<issue 23410>}}
• Now treats YSQL configuration parameters in a case-insensitive manner, avoiding variable duplication and potential failures, aligning closer with Postgres behavior. {{<issue 23478>}}
• Ensures node restarts successfully with secure enabled by introducing a timeout framework. {{<issue 23447>}}
• Restores loading of old dumps that lack index PG class OIDs by enhancing the flag yb_ignore_pg_class_oids. {{<issue 23626>}}
• Promotes cluster start-up when master_webserver_port is customized by considering user input during the validate_and_set_configs step. {{<issue 23629>}}
• Allows the packed row size to exceed its limit during repacking, preventing tserver crash after alter table add column commands with a default value. {{<issue 24050>}}
• The update stabilizes WriteBuffer, preventing potential crashes when constructed over non-zeroed memory. {{<issue 23960>}}
• Adds a null check to prevent a system crash when cleaning up a recently applied transaction state after a flush operation. {{<issue 24026>}}
• Resolves the shutdown issue with the shared exchange causing TServer to hang by properly managing concurrent threads and session destruction, and sets pg_client_use_shared_memory to false by default on Mac to enhance performance. {{<issue 24000>}}
• Ensures "create table" requests for colocated tables don't fail due to mistaken tablet limit checks. {{<issue 23922>}}

CDC

• Refactors CDCSDK to prevent data loss during workload with single shard transactions by ensuring WAL messages are committed before being replicated. {{<issue 19294>}}
• Reduces disk IO overhead by optimizing the tablet-meta update process in CDCService. {{<issue 22805>}}
• Introduces three new yb-admin commands : disable_dynamic_table_addition_in_change_data_stream, remove_user_table_from_change_data_stream, validate_and_sync_cdc_state_table_entries_on_change_data_stream for managing user tables in a CDC stream and addresses a race condition issue {{<issue 22876>}},{{<issue 22835>}},{{<issue 22773>}}
• Obsoletes expired or unpolled tables from CDC stream and state table, reducing unnecessary resource retention and potential blocking of split tablet deletion. {{<issue 23367>}}
• Filters out records related to index tables in colocated databases to prevent GetChanges call failures. {{<issue 23809>}}
• Ensures old-tuples for updates are only sent when replica identity is FULL, matching PostgreSQL behavior. {{<issue 23353>}}
• Enhances table removal from a CDC stream by fetching tablets from the local master copy instead of the entire cdc_state table, enhancing performance. {{<issue 23589>}}
• Prevents crashes that occur due to concurrent GetChanges calls on the same producer tablet. {{<issue 23394>}}
• Ensures entries for dropped tables are removed from the replica identity map, preventing replication errors. {{<issue 24308>}}

yugabyted

• Offers corrected DB OID in the function calculate_table_size, now matching YBCGetDatabaseOid(rel) for improved reliability. {{<issue 20030>}}
• Enables a node to successfully join using the join flag by implementing a retry mechanism. {{<issue 23898>}}

v2.23.0.0 - September 13, 2024 {#v2.23.0.0}

Build: 2.23.0.0-b710

Downloads

Docker:

docker pull yugabytedb/yugabyte:2.23.0.0-b710

Highlights

We're excited to announce the technical preview of four powerful new features in YugabyteDB aimed at simplifying operations, enhancing functionality, and improving performance.

Instant database cloning {{<tags/feature/tp>}}

Instant database cloning allows you to quickly create independent copies of your database for data recovery, development, and testing. Clones can be created in seconds, and initially consume no additional disk space because they share the same data files as the original database. Clones can be created as of now, or as of any time in the recent past (within a configurable history retention period), so developers can experiment without impacting production performance. Clones also provide a safety net for rapid recovery from accidental data loss or corruption.

pg_cron extension {{<tags/feature/tp>}}

We're introducing support for the pg_cron extension, which brings a cron-based job scheduler directly into the database. With pg_cron, you can schedule YSQL commands using familiar cron syntax, including jobs on intervals as fine as seconds. While pg_cron operates on a single node known as the pg_cron leader, the queries it schedules leverage the full resources of the distributed cluster. In case of node failure, leadership is automatically transferred, ensuring continuous availability and smooth scheduling operations.

Semi-automatic xCluster replication {{<tags/feature/tp idea="1186">}}

Semi-automatic xCluster replication simplifies the management of YSQL transactional xCluster replication by operating at the database level instead of the table level. This reduces the need for IT admin involvement when tables are created or dropped, as xCluster management is only required for adding or removing entire databases from replication. DDL operations can be performed by authorized users or DBAs without needing elevated privileges. Semi-automatic xCluster supports YSQL transactional replication and is ideal for Disaster Recovery (DR) purposes.

Improvement to backward scans {{<tags/feature/tp idea="1326">}}

We've improved the performance of backward scans by 10x. For example, descending scans on tables that are stored in ascending order are now much faster. Previously, backward scans were significantly slower than forward scans in YugabyteDB, causing the query optimizer to implement various workarounds, including adding a sort operator on top of the forward scan operator to keep the backward scan pattern comparable. Such optimizer tricks might have been ineffective in complex query plans. The improvements to backward scan performance now allows such queries to be faster out of the box. You enable the feature using the use_fast_backward_scan preview flag.

Change log

Improvements

YSQL

• Enhances logging for DDL transaction conflicts and PG catalog version mismatches by including the DDL command tag and specific log details outside of the log_ysql_catalog_versions flag. {{<issue 20084>}}
• Reduces per-backend memory consumption by reinstating TOAST compression for catalogue tables. {{<issue 21040>}}
• Enables DDL atomicity feature by default by altering ysql_yb_ddl_rollback_enabled, report_ysql_ddl_txn_status_to_master, and ysql_ddl_transaction_wait_for_ddl_verification flags' defaults. {{<issue 22097>}}
• Adds a new YSQL view for YCQL statement metrics, allowing it to be joined with YCQL wait events in the yb_active_universe_history table. {{<issue 20616>}}
• Displays distinct prefix keys explicitly in the explain output, enhancing the clarity of indexing for users. {{<issue 20831>}}
• Adds auto flag ysql_yb_enable_ddl_atomicity_infra to control DDL atomicity feature during the upgrade phase. {{<issue 21535>}}
• Updates read time for each operation to simplify code and avoid applying used read time from obsolete operations. {{<issue 21623>}}
• Allows YbInitPinnedCacheIfNeeded to only load the shared pinned cache, enhancing concurrent handling of DDLs in various databases. {{<issue 21635>}}
• Rectifies a compilation error by eliminating duplicate declaration and unnecessary function triggered by merge issues. {{<issue 20616>}}
• Avoids schema version mismatch errors during ALTER TABLE operations in cases where DDL atomicity is enabled. {{<issue 21787>}}
• Adds new columns to localhost:13000/statements for more comprehensive database management, including user and database IDs along with varied block level statistics. {{<issue 21735>}}
• Now logs global-impact DDL statements that increment all database catalog versions. {{<issue 21826>}}
• Reorganizes extensions into three segregated directories for better access and ease of use. {{<issue 21897>}}
• Resolves schema version mismatch errors that occur after an ALTER TABLE operation due to DDL transaction verification in non-debug builds. {{<issue 21787>}}
• Introduces a new YSQL configuration parameter yb_parallel_range_size for better tuning of parallel range size. {{<issue 21928>}}
• Removes the unused keep_order field from YbctidGenerator for cleaner results processing. {{<issue 21944>}}
• Introduces a new YSQL configuration parameter yb_enable_parallel_append to disable the unannounced feature parallel append. {{<issue 21934>}}
• Performs stylistic modifications and refactors in various YSQL scripts for better readability and performance. {{<issue 22004>}}
• Adds support for creating vector indexes using a dummy ANN method ybdummyann, enabling preliminary vector-based searching in databases. {{<issue 22195>}}
• Restricts the undesired usage of LWFunction by disallowing move copy. {{<issue 22069>}}
• Simplifies the column binding logic for ybgin and lsm access methods and prepares for addition of user-defined index types. {{<issue 22195>}}
• Enables the grammar for CREATE/DROP ACCESS METHOD for more flexible extension handling. {{<issue 22364>}}
• Eradicates misleading log message during table creation with DDL atomicity enabled. {{<issue 22459>}}
• Introduces a new enum PgYbrowidMode to deduplicate hidden internal column addition logic. {{<issue 22536>}}
• Adds Save/Restore state functionality to ConsistentReadPoint using a new Momento class. {{<issue 22597>}}
• Avoids renaming DocDb tables during legacy rewrite operations to prevent issues with backup/restore and improves handling of failed ADD/DROP primary key, ALTER TYPE operations. {{<issue 22802>}}
• Stops python checks on all third-party extensions to avoid build failure. {{<issue 23166>}}
• Simplifies and cleans up code in PgDml/PgSelect/PgSelectIndex classes, ensuring only necessary fields are used and removing redundant destructors, with no logic changes. {{<issue 23192>}}
• Improves backward scans by updating the cost-based optimizer to consider backward scan enhancements, leading to significant execution-time improvements when FLAGS_use_fast_backward_scan is enabled. {{<issue 22370>}}
• Introduces new role-related flags in yb_backup.py script to enhance backup and restore functionalities. {{<issue 20972>}}
• Fixes various issues in the Batch Nested Loop Join code, particularly the new row array comparison, for clearer functioning and better documentation. {{<issue 23407>}}

YCQL

• Now throws an error when using the unsupported GROUP BY clause in YCQL with autoflag ycql_suppress_group_by_error available for compatibility issues. {{<issue 13956>}}

DocDB

• Adds an ABORTED state and an abort_message field to the SysCloneStateInfoPB object for better clone failure management. {{<issue 21054>}}
• Offers a new stack trace tracking framework for improved visibility of disk I/O operations, configurable through the track_stack_traces flag. {{<issue 17993>}}
• Resolves build failure caused by a problematic merge, offering better disk IO visibility by tracking IOs by stacktrace. {{<issue 17993>}}
• Blocks writes based on the total number of bytes being flushed, not just when 2 memtables are flushing. {{<issue 22571>}}
• Allows asynchronous DNS cache updating and resolution retry upon failure to reduce RPC call delays and prevent unexpected leadership changes. {{<issue 22930>}},{{<issue 22311>}}
• Introduces a new flag to toggle on or off recommended memory defaults for increased control over individual memory settings. {{<issue 22161>}}
• Introduces utility to dump top heap stacks when memory limit is exceeded for enhanced debugging. {{<issue 21396>}}
• Deprecates unused flags and old svc_num_workers flags for clearer user configurations. {{<issue 20906>}}
• Allows local debug builds of YugabyteDB to restart from release builds to enhance debugging. {{<issue 21093>}}
• Shifts xCluster-related functions from Catalocustomeranager to XClusterSourceManager for easier management. {{<issue 21325>}}
• Speeds up backward scans by building rows from end to start, reducing unnecessary repositioning and Seek calls, enabled with FLAGS_use_fast_backward_scan flag. {{<issue 19352>}}
• Boosts error messaging clarity when preview flags are not set in allowed_preview_flags_csv. {{<issue 21484>}}
• Prevents GetChangesForXCluster from returning an invalid safe time in certain conditions. {{<issue 21528>}}
• Broadens the rewrite_test_log.py script to incorporate more directory replacements such as home directory, YB_SRC_ROOT, LLVM toolchain directory, and third-party dependencies directory. {{<issue 21532>}}
• Adjusts TServer memory percentage from 50% to 48% for (4,8]GiB boxes and sets new recommendations for boxes over 16 GiB. {{<issue 20664>}}
• Incorporates the usearch and fp16 header-only libraries into the src directory, simplifying code import. {{<issue 21830>}}
• Returns the original error messages from CreateTable and CreateTablegroup instead of an incorrect "Database not found" message. {{<issue 21760>}}
• Enables replication of DDL schemas and users to maintain consistency between source and target. {{<issue 21848>}}
• Updates AreNodesSafeToTakeDown to return earlier and deliver a readable error message when hitting a timeout, using a control flag, are_nodes_safe_to_take_down_timeout_buffer_ms with a default setting of 2 seconds. {{<issue 21855>}}
• Reduces unnecessary alerts by removing "No active snapshot" warning from the logs. {{<issue 21911>}}
• Ensures Data Definition Language (DDL) operations replicate exactly once by checking the replicated_ddls table prior to rerunning any DDL. {{<issue 21943>}}
• Streamlines the creation of xCluster streams by unifying the scattered code into XClusterClient::CreateXClusterStream<Async>. {{<issue 22343>}}
• Allows for faster failover in xCluster DR by skipping the cleanup process when skip_producer_stream_deletion is set on DeleteUniverseReplicationRequestPB. Safe for upgrades and rollbacks. {{<issue 22050>}}
• Reduces extra reactor threads by reusing server messenger in AutoFlags. {{<issue 22076>}}
• Reduces resource usage by enabling stateful service client to reuse the server's existing yb_client. {{<issue 22102>}}
• Adds SCHECK_PB_FIELDS_NOT_EMPTY macro for validating non-empty fields, with individual checks on list elements. {{<issue 22182>}}
• Splits up yb_xcluster_ddl_replication.c into additional util files for better project extensibility. {{<issue 22190>}}
• Removes deprecated JSON output format in xCluster for clean and efficient function. {{<issue 22219>}}
• Allows only single DDL query strings to prevent issues with DDL+DML mixes or multiple DDLs. {{<issue 22060>}}
• Shifts certain RPC endpoint methods from the Catalocustomeranager to the new MasterClusterHandler class for easier management. {{<issue 19715>}}
• Adds read-time option description in the help of ysql_dump allowing database dump at a specified past time. {{<issue 21886>}}
• Renames and replaces cdc_consumer_handler_thread_pool_size with xcluster_consumer_thread_pool_size to reduce CPU and memory usage. {{<issue 20305>}}
• Refreshes stack trace tracking UI endpoints to enable per-column sorting and optimize sorting script. {{<issue 22841>}}
• Allows table locking by acquiring local server object locks for DDLs and DMLs, hosted by a local transaction participant, with session ID and tied to their lifetime. Includes performance upgrades to lock acquisition and conflict resolution. {{<issue 23045>}}
• Simplifies replication setup by using table IDs instead of names, helping avoid issues caused by table renames or recreations. {{<issue 23013>}}
• Increases speed of backward scans for flat document reader with support for packed row V2. {{<issue 22556>}}
• Reuses the Tservers yb::client in CDCService to decrease 4 threads and a meta cache, addressing a CDCService/xCluster source issue. {{<issue 22893>}}
• Allows unified usage of XClusterRemoteClient in XClsuterConsumer, centralizing client creation logic. {{<issue 22908>}}
• Ensures replication health before succeeding IsSetupUniverseReplicationDone, improving error detection. {{<issue 22948>}}
• Adds external_hybrid_time to log-dump output to detect xCluster target writes. {{<issue 22918>}}
• Introduces two new SOURCE_UNREACHABLE and SYSTEM_ERROR enums to enable more detailed error reporting from the Poller. {{<issue 22996>}}
• Allows requesting streams by producer table ids for xCluster DDL Replication to accurately match tables together. {{<issue 23013>}}
• Enhances stack trace tracking endpoints usability by adding an access button, a reset tracking link, and timestamp details. {{<issue 22842>}}
• Renames and shifts various members and functions from Catalocustomeranager to XClusterManager. {{<issue 23044>}}
• Replaces the deprecated and unused namespace replication with DB Scoped replication for a more efficient solution. {{<issue 23046>}}
• Makes pggate aware of fast backward scan capability for accurate cost identification in the Cost Based Optimizer. {{<issue 22937>}}
• Added emergency_repair_mode flag and new yb-admin commands to handle corrupted CatalogEntity data without stopping yb-master. {{<issue 23098>}}
• Relocates Setup, Bootstrap, Alter and Delete Target Replication functions for better organization. {{<issue 23183>}}
• Changes column ID representation in debug builds to be compatible with release builds. {{<issue 21093>}}
• Refactored the PopulateTabletCheckPointInfo function to improve its readability and maintainability. {{<issue 23301>}}
• Removes display of InvalidFlags in the flags UI to clean up the user interface. {{<issue 23308>}}
• Allows for storing and loading of vector indexes, ensuring effective management of these indexes. {{<issue 23377>}}
• Integrates class DocDBStatistics with ReadOperationData for better statistics transmission to IntentAwareIterator. {{<issue 23420>}}
• Introduces a new flag max_disk_throughput_mbps for automated control of write rejections when disk is full, replacing reject_writes_min_disk_space_aggressive_check_mb. {{<issue 23373>}}
• Adds a flag FLAGS_tablet_split_min_size_ratio to control tablet splitting based on SST file sizes, ensuring better control over tablet size imbalance. {{<issue 21458>}}
• Relocates heartbeat code for greater readability and Catalog Manager size reduction. {{<issue 21899>}},{{<issue 19715>}}
• Simplifies the clone state manager by moving persisted data to an in-memory structure. {{<issue 22138>}}
• Relocates specific heartbeat code to master_heartbeat_service.cc for enhanced readability and easier tracking. {{<issue 21899>}},{{<issue 19715>}}
• Refactors heartbeat path code for easier navigation and reduction of Catalocustomeranager size, with no functional changes. {{<issue 21899>}},{{<issue 19715>}}
• Shifts tcmalloc profiling code to the Utils folder. {{<issue 22258>}}
• Moves tablet splitting specific RPCs and functions from catalog_manager.cc to tablet_split_manager.cc for better code management. {{<issue 22603>}},{{<issue 19715>}}
• Introduces a new flag enable_rwc_lock_debugging to control slow lock debugging and fixes a bug in rwc_lock.cc. {{<issue 22807>}}
• Changes CloneStateInfo object from scoped_refptr to std::shared_ptr for standardization. {{<issue 23036>}}
• Allows setting only non-empty schema names in YBTableName. {{<issue 23371>}}
• Transfers ownership of TabletSplitManager, CloneStateManager, and SnapshotCoordinator from Catalocustomeranager to Master for leaner dependency requirements. {{<issue 22603>}},{{<issue 19715>}}

CDC

• Preserves CDC stream even when all associated tables are dropped, tying its lifecycle to the database. {{<issue 21419>}}
• Introduces three new yb-admin commands to remove a user table from a CDCSDK stream, descend dynamic table addition in a CDC stream, and validate CDC state for a particular stream, enhancing control over CDC streams. {{<issue 22876>}},{{<issue 22773>}}
• Prevents addition of tables with enum array column to the CDC stream to avoid crashes during consumption. {{<issue 22897>}}
• Transforms the flag yb_enable_cdc_consistent_snapshot_streams from a preview into a default true auto flag. {{<issue 22984>}}
• Enables dynamic table addition with Postgres replication consumption by setting retention barriers on new tables' tablets during creation. {{<issue 21643>}}
• Allows modification of the publication refresh interval using the cdcsdk_publication_list_refresh_interval_secs flag. {{<issue 21796>}}
• Adds a TServer flag, ysql_yb_default_replica_identity, for customizing default replica identity at table creation. {{<issue 22326>}}
• Introduces cdcsdk_enable_dynamic_table_addition flag to manage dynamic table additions in replication slot consumption model. {{<issue 22406>}}
• Introduces replication slot name for internal distinction between two consumption models in the code. {{<issue 22810>}}
• Allows creating an old model stream via yb-admin in upgraded environments, ensuring only one stream type per database. {{<issue 22894>}}
• Introduces replica identity in CDC to populate before image records, allowing table-level before image information fetching and retaining in stream metadata. {{<issue 21314>}}
• Eliminates unnecessary NOTICE messages when setting yb_read_time from walsender, reducing message clutter. {{<issue 22379>}}
• Enables transaction state to be cleared promptly after a table is deleted, preventing table deletion from getting stuck and resulting in faster functionality. {{<issue 22095>}}

yugabyted

• Allows Connection Manager to handle error messages, preventing test failures in the YSQL layer. {{<issue 21756>}}
• Ensures RENAME DATABASE query handles logical and physical connections correctly for consistent database behavior. {{<issue 21284>}}
• Redefines tracking of role modifications using role OID in YSQL Connection Manager for accurate behavior during mid-session role renaming. {{<issue 21505>}}
• Offers support for single-use YSQL configuration parameters in YSQL Connection Manager. {{<issue 21757>}}
• Enables using role OID with session_authorization in YSQL Connection Manager for correct role alterations in a session. {{<issue 21637>}}
• Allows sticky connections when setting certain YSQL configuration parameters not permitted in explicit transactions. {{<issue 22957>}}
• Allows modification of YSQL configuration parameters on a running cluster by destroying the control connection. {{<issue 21516>}}
• Adds a new /pitr endpoint and screen in the database page to list scheduled PITRs on yugabyted UI. {{<issue 21355>}}
• Introduces upgrade_finalize command for smoother YugabyteDB version upgrades using yugabyted CLI and includes an upgrade_ysql_timeout flag. {{<issue 21888>}}
• Directly enables yb_enable_read_committed_isolation and ysql_enable_read_request_caching on yb-master and yb-tserver processes. {{<issue 22061>}}
• Delivers alerts on user interface when encountering node version mismatches in the cluster. {{<issue 21888>}}
• Simplifies yugabyted by dropping Python2 support and transitioning the script to use Python3, replacing deprecated distutils package with shutil. {{<issue 22072>}},{{<issue 21409>}}
• Enables better handling of multi-valued flags in yugabyted without duplication, making the system more maintainable. {{<issue 22091>}}
• Corrects the Sankey diagram for CPU usage by accurately calculating the total number of used/available cores. {{<issue 22125>}}
• Made changes to string literals in yugabyted to avoid SyntaxWarning and added checks for exceptions during incorrect advertise_address input. {{<issue 22210>}},{{<issue 22230>}}
• Enables correct parsing of startup parameters with spaces in values when using the YSQL Connection Manager. {{<issue 22248>}}
• Allows to specify multiple data directories using the new additional_data_dir configuration. {{<issue 22126>}}
• Enables xCluster replication management between database clusters using new yugabyted commands. {{<issue 22349>}}
• Ensures yugabyted UI metrics display properly with Kubernetes OSS operator deployed clusters. {{<issue 22532>}}
• Elevates the reliability of the UUID retrieval process for tablet server nodes on the user interface. {{<issue 22532>}}
• Ensures accurate CPU usage metrics by updating prev_ticks_ at each metrics snapshotting iteration. {{<issue 22910>}}
• Allows smooth node restart even if the data_dir parameter is missing in the user configuration file. {{<issue 23052>}}
• Reduces collect_logs command failures by removing the yugabyted running check even when the yugabyted process is not running. {{<issue 23210>}}
• Enhances yugabyted configure_read_replica commands with checks to gracefully handle failures when data_placement_constraint lacks :<num_of_replicas>. {{<issue 23273>}}

Bug fixes

YSQL

• Fixes an error that occurs when decoding null values from a boolean column sorted as NULLS LAST in a secondary index. {{<issue 22121>}}
• Fixes YSQL upgrade single connection mode error preventing new connection attempts before the old ones are released. {{<issue 22283>}}
• Allows YB Admins to run pg_locks without requiring superuser status. {{<issue 23266>}}
• Avoids failure when upgrading from version 2.14/2.16 to 2.20 by introducing a check to ensure pggate can handle RPC metrics sidecar before sending Scanned Rows count. {{<issue 21229>}}
• Fixes memory leaks in pg_constraint/pg_attrdef local cache by adding a missing hash_destroy call in YbCleanupTupleCache. {{<issue 22262>}}
• Resolves remaining memory leaks in CacheMemoryContext to stabilize cache memory after every catalog cache refresh. {{<issue 22262>}}
• Allows YSQL DDL operations to wait for rollback/roll-forward operations to finish before proceeding. {{<issue 20033>}}
• Allows more accurate modeling of base scan costs by taking into account the impact of storage index filters on secondary index. {{<issue 20635>}}
• Resolves colocation option issues in table creation linked to table rewriting and partitioning. Enhances the defGetBoolean function to parse string values "0" and "1" as false and true respectively, and shifts a verification step to a earlier spot in the CREATE TABLE execution path, ensuring successful table partition creation. {{<issue 20302>}},{{<issue 20914>}}
• Renames the YSQL configuration parameter ddl_rollback_enabled to yb_ddl_rollback_enabled for specificity. {{<issue 21480>}}
• Adds network latency cost to startup cost, yielding more accurate cost calculations in small tables. {{<issue 20898>}}
• Disables bitmap scan by default to prevent unwarranted selection due to lower CBO costs. {{<issue 21479>}}
• Reduces unnecessary log messages when catalog_version_table_in_perdb_mode is set to true. {{<issue 21481>}}
• Corrects an issue where certain unbatchable filters weren't detected during indexpath formation when indexpath accepted batched values from multiple relations. Requires backports to 2.20 and 2.18. {{<issue 21292>}}
• Exposes the YSQL configuration parameter yb_enable_optimizer_statistics as a flag ysql_yb_enable_optimizer_statistics. {{<issue 21650>}}
• Corrects buffer overflow during placement validation in ALTER TABLE SET TABLESPACE operation. {{<issue 21655>}}
• Allows for an enhanced readability and performance of yb_cost_index code, aiding in merging with the pg15 branch. {{<issue 21672>}}
• The deadlock issue occurring when both a table and its index are deleted concurrently in yb-master has been resolved. {{<issue 21663>}}
• Refines the YbGetOrdinaryColumnsNeedingPgRecheck condition to align with the ybIsTupMismatch implementation, ensuring Postgres rechecks index conditions when the "preliminary check" is skipped due to an invalid target key attnum. {{<issue 21451>}}
• Corrects checks in YbIsScanCompatible to ensure the right-hand side (RHS) of all bound index conditions, not just inequalities, fits into the left-hand side (LHS) datatype. {{<issue 21758>}}
• Prevents query layer retries for multi-statement queries to avoid redoing whole queries, ensuring idempotence. {{<issue 21361>}}
• Fixes a bug that caused incorrect setting of global catalog version mode on TServer start. {{<issue 21850>}}
• Fixes a bug in the index tuple width calculation for better YB base scans cost model. {{<issue 21892>}}
• Ensures pushed down RowCompareExpressions correctly enforce non-null column references, rectifying previous behavior and enhancing data accuracy. {{<issue 21847>}}
• Reduces the frequency of schema version mismatch errors during consecutive DDL operations by ensuring the up-to-date schema is fetched. {{<issue 21706>}}
• Allows usage of YsqlDdlRollbackEnabled in pggate C++ code by correctly passing the result of YbDdlRollbackEnabled, reducing DDL atomicity g-flag issues in RF 1 clusters. {{<issue 21706>}}
• Adds a new flag ysql_min_new_version_ignored_count to prevent a TServer crash caused by the downward shift in yb-master's catalog version, often surfacing post a PITR restore operation. {{<issue 21776>}}
• The postgres process no longer crashes when running a "show all" command due to correct placement of the yb_enable_ddl_atomicity_infra description. {{<issue 21947>}}
• Prevents failures in transaction restarts with UPDATE ...RETURNING queries in debug builds. {{<issue 22010>}}
• Reverts updates from Storage SQL to Remote SQL and Storage Filter to Remote Filter for Foreign Scan. {{<issue 22070>}}
• Adjusts inaccurate ALTER TABLE rewrite check for dropped rules to prevent unnecessary command failure. {{<issue 22064>}}
• Fills in the "relation" column in pg_locks with the correct table OID after a table rewrite. {{<issue 22081>}}
• Adjusts the value of YB_AT_REWRITE_ALTER_PRIMARY_KEY to prevent flag clashes and accommodate future upstream PG flags. {{<issue 22086>}}
• Fixes the issue of PG crash when yb_debug_log_catcache_events=1 is used before a database has been selected. {{<issue 22139>}}
• Enables backward parallel scan capabilities, adjusting key bounds when conducting descending order scans. {{<issue 21633>}}
• Prevents unnecessary CPU cycles and log flooding by not reading pg_yb_catalog_version when enable_ysql=false. {{<issue 22213>}}
• Corrects the log message for successful column drop operations, ensuring accurate representation of DDL operations. {{<issue 22243>}}
• Stops Batched Nest Loop (BNL) crashes by ensuring better indexing condition checks. {{<issue 21954>}}
• Refines the logic to accurately push down join clauses to batched index scans without causing conflicts. {{<issue 21878>}}
• Grants BNL hashtable its own expression context to prevent data overwrites during query execution. {{<issue 21266>}}
• Re-enables rechecking for RowCompareExpressions to accurately handle NULL inputs in scan bound calculations. {{<issue 22075>}}
• Resolves the old-style-declaration error in YbDdlRollbackEnabled by changing its definition to static inline bool. {{<issue 22334>}}
• Prevents potential crashes by ensuring yb_table_properties pointer, in load_relcache_init_file, does not point to random, invalid memory. {{<issue 22342>}}
• Makes yb_get_range_split_clause robust using PG TRY CATCH block, ensuring YB backup doesn't fail. {{<issue 22356>}}
• Fixes memory leaks in ybcFetchNextHeapTuple by properly freeing the YBCStatus. {{<issue 22396>}}
• Prevents core dumps by ensuring YSQL webserver destruction upon receiving a termination signal. {{<issue 18948>}}
• Introduces new functions to enhance and consolidate the focus on tables stored in the system catalog. {{<issue 22520>}}
• Allows CreateNamespaceIfNotExists function to retry on "already exists" error, preventing race conditions. {{<issue 22512>}}
• Fixes the issue when a separately created and later attached partition does not properly inherit the parent's primary key using ALTER TABLE ...ATTACH PARTITION. {{<issue 22562>}}
• Resolves potential database OID collision with system_postgres by excluding reserved OID 65535 in allocation. {{<issue 22598>}}
• Allows skipping the relfilenode check on parent partition tables which do not get recreated during table rewrites. {{<issue 22625>}}
• Now allows for correct backward prefix-based scanning by eliminating the problematic kGroupEnd marker that was leading to inaccurate seek results. {{<issue 22615>}}
• Removes the unused function Catalocustomeranager::WaitForDdlVerificationToFinish for clarity. {{<issue 22649>}}
• Ensures bitmap scans correctly recheck all results and avoid excluding rows, improving accuracy of outcomes. {{<issue 22622>}}
• Prevents the IN expressions on single column from wrongly taking the tuple path, ensuring correct data processing. {{<issue 22704>}}
• Fixes incorrect access to the scan plan's bind descriptor during tuple IN condition rechecks. {{<issue 22800>}}
• Allows the creation of new shared relations during YSQL upgrade to have a global impact by incrementing the catalog version across every database. {{<issue 22830>}}
• Allows resetting of stats collected by the ANALYZE command, including reltuples, pg_statistic rows, and pg_statistic_ext values. {{<issue 22028>}}
• Corrects the YbGetOrdinaryColumnsNeedingPgRecheck function to return table column numbers instead of index numbers, preventing unnecessary data fetches and potential crashes or errors after dropping a column. {{<issue 22832>}}
• Resolves a detected deadlock during ALTER TABLE operations, enhancing test stability. {{<issue 22882>}}
• Reduces unexpected log messages by not invoking YsqlDdlTxnCompleteCallback if all table 'pb_txn_id's in the DDL transaction verifier state are already cleared, avoiding potential deadlock situations in DDL atomicity. {{<issue 22882>}}
• Reduces prefix length in the index when using distinct index scan with included columns. {{<issue 22822>}}
• Returns more accurate results when running EXPLAIN command by fixing relids of prefix keys under a subquery distinct index scan. {{<issue 22923>}}
• Reduces sequence cache collision by incorporating both database and sequence OIDs as the entry key. {{<issue 22935>}}
• Prevents a crash related to memory release associated with TupleTableSlots in SubPlans during a Values Scan. {{<issue 22967>}}
• Allows faster data inserts into tables with identity columns. {{<issue 22837>}}
• Enhances log output by adding missing newlines in the yb_pclose_check function and corrects memory allocation. {{<issue 23057>}}
• Streamlines the "drop column" operation process, preventing hindrance even if the alter schema RPC is missed. {{<issue 23100>}}
• Reduces ASAN/TSAN builds' pressure on t-server/master and avoids timeout issues by using less parallelism. {{<issue 22594>}}
• Eliminates unnecessary waiting for concurrent transactions in the DEFERRABLE mode for READ ONLY serializable transactions. {{<issue 23120>}}
• Fixes TServer crash when pushing down certain SAOP operations like string_to_array. {{<issue 23287>}}

YCQL

• Allows the deletion of the Cassandra role in YCQLsh without it regenerating upon cluster restart, by adding a flag to mark if the role was previously created. {{<issue 21057>}}
• Removes extra reads during the processing of INSERT INTO ...RETURNS STATUS AS ROW for CQL tables with a specific primary key, improving system load and efficiency. {{<issue 23330>}}
• Now ensures simultaneous registration of new split tablet children, fully covering the keyspace during splits. {{<issue 19954>}}

DocDB

• Fixes hidden split parent tablets wrongly appearing as leaderless in the master's leaderless tablet endpoint, ensuring accurate load balance status and preventing potential issues with Point-in-Time Recovery (PITR) operations. {{<issue 21371>}}
• Ensures failed xCluster setup if the xCluster stream state update to ACTIVE does not occur. {{<issue 22601>}}
• Fix ensures heartbeat processing doesn't blindly overwrite tablet replica state, avoiding potential data corruption. {{<issue 21836>}}
• Fixes crash when parsing an invalid timestamp in LTO build by updating C++ dependencies and error handling. {{<issue 22191>}}
• Removes unnecessary flush during snapshot deletion, preventing write blocks. {{<issue 22369>}}
• Eliminates possible deadlock during setup replication by fixing the order in which locks are acquired. {{<issue 22376>}}
• Fixes issue of scans not honoring timeouts, preventing indefinite reads and reducing CPU usage. {{<issue 21829>}}
• Allows restoring a snapshot schedule from a time just before the oldest snapshot, improving usability. {{<issue 21269>}}
• Removes assumption that every tablet server hosts tablets, preventing potential crashes. {{<issue 20230>}}
• Resolves a heartbeat metrics issue ensuring full xCluster error information is sent to the new master even during a leader failover, and makes tserver_heartbeat_metrics_interval_ms runtime updatable. {{<issue 22624>}}
• Adds validation to RPCs DeleteSnapshot and RestoreSnapshot to prevent deletion or use of snapshots involved in ongoing processes. {{<issue 23055>}}
• Ensures Create Table operation fails if Alter Replication encounters an error, enhancing the reliability of replication setup. {{<issue 21732>}}
• Converted the ysql_skip_row_lock_for_update to an auto-flag to resolve compatibility issues during upgrade, preventing incorrect DB record creations that can affect row visibility and integrity. {{<issue 22057>}}
• Modifies memory consumption calculations for pending operations to ensure accurate rejection of new writes at bootstrap, preventing loading failures. {{<issue 21254>}}
• Trims large error messages in AsyncRpc::Failed to prevent hitting memory limit and resulting unavailability. {{<issue 21402>}}
• Excludes hidden tables from generate snapshot output to circumvent cloning failure from recreated tables. {{<issue 21631>}}
• Prevents unexpected leader fatal errors by updating cached leader terms immediately after a leader change. {{<issue 21808>}}
• Renames and updates the description of the flag min_secustomerent_size_to_rollover_at_flush for clarity. {{<issue 21691>}}
• Changes the class of enable_automatic_tablet_splitting flag from kLocalPersisted (class 2) to kExternal (class 4) to eliminate setup issues with xCluster configurations. {{<issue 22088>}}
• Switches from using scoped_refptr to std::shared_ptr for TabletInfo to handle cycles safely. {{<issue 18257>}},{{<issue 21139>}}
• Updates cotable IDs in flushed frontiers during a snapshot restore, preventing potential post-restore issues. {{<issue 23047>}}
• Allows the persistent mapping of source-target schema versions when a new table is added to a colocated database, thus preventing replication from stalling after T-server restarts. {{<issue 23188>}}
• Eliminates potential FATAL errors during reported tabletPB creation by ensuring retrieval of schema version is atomic. {{<issue 21340>}}
• Ensures the correct order of destroying components, preventing possible concurrent calls on a WAL append callback. {{<issue 21564>}}
• Adds a TSAN suppression to manage the apparent race condition in the function boost::regex_match. {{<issue 21585>}}
• Fixes the compilation error for almalinux8 fastdebug gcc11 that was previously removed from the build matrix. {{<issue 21536>}}
• Corrects a bug causing some tablet metrics to display incorrect metric_type attribute. {{<issue 21608>}}
• Fixes a segmentation fault in yb-master by checking for a null pointer before dereferencing it, addressing an issue in the CDC run on 2.23.0.0-b37-arm. {{<issue 21648>}}
• Reduces unnecessary logging during checkpoint operations by lowering INFO level logs to DEBUG_LEVEL, enhancing log readability. {{<issue 21658>}}
• Allows DML operations on non-replicated databases and blocks DML only on databases in transactional xCluster replication STANDBY mode. Now only databases part of an inbound transactional xCluster replication group in the xCluster safe time map will have DML operations blocked. Also, certain attributes are moved from TServer to TserverXClusterContext. {{<issue 21245>}}
• Enables the session to outlive the callback by holding a shared pointer to it, preventing potential crashes during concurrent DML queries. {{<issue 21103>}}
• Avoids multiple destruction of the same database connection, preventing system crashes due to simultaneous connection failures. {{<issue 21738>}}
• Allows viewing of the RPC bind addresses in the master leader UI, especially beneficial in cases like k8s where the RPC bind address with the pod DNS is more useful than the broadcast address. {{<issue 21959>}}
• Prevents fatal errors by skipping ReserveMarker/AsyncAppend if the tablet peer has already been shut down. {{<issue 21769>}}
• Prevents yb-master crash by ensuring background task isn't deleted before the callback is invoked. {{<issue 21773>}}
• Enables callback completion wait in PollTransactionStatusBase during shutdown to prevent unexpected process termination. {{<issue 21773>}}
• Initializes prev_op to UNKNOWN to prevent AlmaLinux 8 fastdebug gcc11 compilation failures. {{<issue 21811>}}
• Enables batched metric updates for YCQL reads to prevent performance drop due to RocksDB metric updates. {{<issue 21832>}}
• Removes pending delete logic from load balancer to prevent delays during high tablet replica movement. {{<issue 21806>}}
• Enhances YSQL operation by refining task shutdown procedures and avoiding unnecessary task aborts. {{<issue 21917>}}
• Stops fatal errors caused by the re-use of remote log anchor session during remote bootstrap from a non-leader peer. This fix ensures shared pointers are accurately tracked for tablet_peer objects using the = operator, preventing unintentional destruction of underlying objects. {{<issue 22007>}}
• Delays min_running_ht initialization until after the successful completion of tablet bootstrap to prevent unexpected behaviors. {{<issue 22099>}}
• Enables the skip_table_tombstone_check for colocated tables to prevent errors. {{<issue 22115>}}
• Prevents potential segfaults during catalog reload by modifying GetClusterConfig function. {{<issue 21775>}}
• Reduces the interval of the tablet server metrics heartbeat to prevent potential misreporting of a leaderless tablet. {{<issue 22189>}}
• Reduces four threads and a meta cache in xCluster consumer by reusing the TServers yb::client. {{<issue 22845>}}
• Resolves the issue of pg_locks query failure due to missing host node UUID in distributed transactions. {{<issue 22181>}}
• Clarifies memory division flags to reflect they are percentage of the process's hard memory limit, not total available memory. {{<issue 22423>}}
• Eliminates latency spikes in conflicting workloads by preventing redundant ProbeTransactionDeadlock RPCs. {{<issue 22426>}}
• Corrects the CI build issues on GCC 12, debug AlmaLinux 9 caused by updates in cf0c09b. {{<issue 22501>}}
• Captures the actual user executing the query instead of only the superuser and fixes ordering in the ddl_queue handler. {{<issue 22514>}}
• Enhances logging during MemTable flushing to better monitor memory usage limits. {{<issue 22737>}}
• Prevents premature metric destruction during Prometheus scrapes, resolving non-UTF8 character issues. {{<issue 22767>}}
• Ensures object drops correctly cascade to dependent columns in the DocDB table, preventing inconsistencies. {{<issue 22874>}}
• Boosts YSQL major version upgrade process by minimal changes in the master branch to reduce its divergence. {{<issue 23079>}}
• Adds a flag to disable the intent filtering during bootstrap, preventing potential data corruption on restart. {{<issue 23184>}}
• Allows large bytes to be requested on RateLimiter, preventing indefinite call stalling. {{<issue 23173>}}
• Restores the previously missing home icon in the master user interface. {{<issue 23275>}}
• Removes the read-time option from the ysql_dump help output for proper DDL atomicity handling. {{<issue 23299>}}
• Allows setting custom snapshot retention duration using yb-admin, including retaining a snapshot indefinitely by setting retention_duration_hours to 0. {{<issue 23332>}}
• Removes an unreachable line causing GH build failure. {{<issue 23375>}}
• Corrects an issue where the load balancer improperly handles a pending leader stepdown task. {{<issue 21834>}}
• Removes the TServer warning log for mismatching cluster config versions, reducing unnecessary noise in logs. {{<issue 23119>}}
• Eliminates memory leaks in YSQL Connection Manager by ensuring proper deallocation of objects and variables. {{<issue 10065>}}
• Resolves the TServer crash issue during query execution happening due to a NULL pointer dereference. {{<issue 23004>}}
• Allows for a reliable connection to a remote YugabyteDB universe by setting skip_master_flagfile when creating YBClients. {{<issue 23145>}}
• Ensures the node restarts properly even with secure mode enabled by adding a timeout framework. {{<issue 23447>}}
• Ensures the stack_is_too_deep function returns predictable results in ASAN, aiding effective limit setting on stack depth. {{<issue 15682>}}
• Introduces a check for multi-threaded mode in catalog lookup functions to avoid server crashes and make error mitigation easier for users. {{<issue 23401>}}
• Enhances visibility of the Hidden state in Master/Tserver Tables UI by shifting its position more prominently to the State column. {{<issue 22521>}}
• Increases the speed of the yb-admin snapshot schedule create command to reduce resource usage. {{<issue 21929>}}
• Deprecates the TServer flag enable_pg_savepoints to prevent incorrect behavior and avoid silent progress in PL/pgSQL exceptions. {{<issue 29018>}}
• Eliminates the occurrence of "schema version mismatch" error following a DROP INDEX statement by introducing a delay in index deletion. {{<issue 22637>}}

CDC

• Ensures deletion of MemoryContext after each GetChanges RPC to prevent memory leaks. {{<issue 22328>}}
• Introduces additional VLOG statements in the ListReplicationSlots function for better debugging. {{<issue 21652>}}
• Prevents newly created indexes, materialized views, and non-user tables from being added to the Chang Data Capture (CDC) stream metadata. {{<issue 22808>}}
• Reduces resource usage by removing non-eligible tables, like indexes, from existing CDC SDK stream metadata, and releasing retention barriers. This change requires the master flag enable_cleanup_of_non_eligible_tables_from_cdcsdk_stream and limits processing to two non-eligible tables per namespace per run with cdcsdk_table_processing_limit_per_run. Introduces three yb-admin commands for managing CDC streams. {{<issue 22876>}},{{<issue 22835>}},{{<issue 22773>}}
• Introduces new auto flag cdcsdk_enable_identification_of_non_eligible_tables and three yb-admin commands to manage tables in CDC stream, enhancing control and reducing unnecessary resource usage. {{<issue 22876>}},{{<issue 22835>}},{{<issue 22773>}}
• Fixes the issue of getting either 0 or a random time as the server's system clock in XLogData from the logical replication stream. {{<issue 22929>}}
• Fixes a memory leakage issue in the walsender process by deep freeing the cached record batch after streaming to the client. {{<issue 21530>}}
• Adds more debug logs in the walsender to aid in investigating issues like linked data loss. {{<issue 21465>}}
• Allows for better memory management in the walsender process by storing record batches in a separate memory context. {{<issue 21530>}}
• Logs RPC errors as warnings during the cleanup of virtual WAL after LogicalReplication ends. {{<issue 21651>}}
• Allows stream replication to handle serialized transactions successfully by adding directory creation logic. {{<issue 21765>}}
• Stops loading replication slots from disk during startup to avoid potential system crashes. {{<issue 21841>}}
• Adds more logs for easy debugging during stress runs for Change Data Capture (CDC) without any impact on performance. {{<issue 21780>}}
• Limits unnecessary RPC calls to the local TServer during RollbackToSubTransaction operation if transaction is read-only, a fast-path transaction, or has NON_TRANSACTIONAL isolation level. {{<issue 21519>}}
• Limits transactions' inclusion in the unacked list only upon receiving the commit record, enhancing the restart_lsn calculation. {{<issue 21950>}}
• Removes table level attributes from CDCSDK metrics to avoid TServer crash due to failed DCHECK assertion. {{<issue 22142>}}
• Fixes the segmentation fault in walsender for dynamic table addition by refreshing stored replica identities and preventing a race condition when creating dynamic tables. {{<issue 22273>}}
• Updates the serialization and de-serialization logic to include the yb_is_omitted array, preserving data values in large transactions. {{<issue 21946>}}
• Solves an issue where CDCSDK incorrectly deduces tablets as not interesting for stream before reaching the configured time limit. {{<issue 22383>}}
• Addresses a race condition in dynamic table creation, enhancing stability during table and tablet initialization. {{<issue 22408>}}
• Refines the logic to remove BEGIN record when no DML records are added, preventing potential virtual WAL crashes. {{<issue 21646>}}
• Resolves "could not open relation" error by updating slot creation method and simplifying yb_read_time logic. {{<issue 22398>}}
• Enables support for dynamically allotted OID data types in CDC to prevent system crashes. {{<issue 23179>}}
• Allows handling of non-eligible table cleanup in CDC stream loading even after table drop, preventing master crash. {{<issue 23278>}}
• Reduces total inserts from 5k/thread to 2.5k/thread for clearer consumption of expected records. {{<issue 23128>}}
• Prevents failures in decoding change events by refreshing cached_schema_details when executing a new GetChanges request if the client indicates a necessity for the schema. {{<issue 20698>}}
• Allows pg_replication_slots to return an empty response instead of an error when ysql_yb_enable_replication_commands flag is false. {{<issue 23096>}}