What's new in the YugabyteDB v2.18 STS release series

Release announcements

• Introducing YugabyteDB 2.18: Unlock Kubernetes and Simplify Operations

Release notes

{{< warning title="Upgrade to release 2.18.7.0 or later" >}}

To avoid a rare race condition that can be encountered in v2.18.5-2.18.6, upgrade to release v2.18.7.0 or later.

{{< /warning >}}

What follows are the release notes for the YugabyteDB v2.18 release series. Content will be added as new notable features and changes are available in the patch releases of the YugabyteDB v2.18 release series.

v2.18.9.0 - August 15, 2024 {#v2.18.9.0}

Build: 2.18.9.0-b17

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.9.0-b17

Improvements

YSQL

• Ensures failed ADD/DROP pkey, ALTER TYPE operations no longer leave DocDB table renamed to <table_name>_temp_old, avoiding backup/restore issues. {{<issue 22802>}}

DocDB

• Allows asynchronous DNS cache updating and resolution retry upon failure to reduce RPC call delays and prevent unexpected leadership changes. {{<issue 22930>}},{{<issue 22311>}}

Bug fixes

YSQL

• Prevents the IN expressions on single column from wrongly taking the tuple path, ensuring correct data processing. {{<issue 22704>}}
• Reduces sequence cache collision by incorporating both database and sequence OIDs as the entry key. {{<issue 22935>}}
• Prevents a crash related to memory release associated with TupleTableSlots in SubPlans during a Values Scan. {{<issue 22967>}}

DocDB

• Adds validation to prevent deletion or restoration of snapshots in use by other operations. {{<issue 23055>}}
• Prevents inconsistencies between DocDB and PG by ensuring column drops cascade correctly when dependent objects are dropped. {{<issue 22874>}}
• Allows large byte requests on RateLimiter by breaking them into multiple smaller requests, ensuring smooth xCluster operation. {{<issue 23173>}}

CDC

• Prevents newly created indexes, materialized views, and non-user tables from being added to the Change Data Capture (CDC) stream metadata. {{<issue 22808>}}

v2.18.8.1 - July 18, 2024 {#v2.18.8.1}

Build: 2.18.8.1-b3

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.8.1-b3

Improvements

DocDB

• Allows asynchronous DNS cache updating and resolution retry upon failure to reduce RPC call delays and prevent unexpected leadership changes. {{<issue 22930>}},{{<issue 22311>}}

v2.18.8.0 - June 21, 2024 {#v2.18.8.0}

Build: 2.18.8.0-b42

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.8.0-b42

Improvements

YSQL

• Introduces yb_silence_advisory_locks_not_supported_error as a temporary solution for users to avoid disruption when using advisory locks without actual lock acquisition. {{<issue 19974>}}
• Adds new columns to localhost:13000/statements for more comprehensive database management, including user and database IDs along with varied block level statistics. {{<issue 21735>}}

YCQL

• Now throws an error when using the unsupported GROUP BY clause in YCQL with autoflag ycql_suppress_group_by_error available for compatibility issues. {{<issue 13956>}}

DocDB

• Solves the potential deadlock in DeleteTabletListAndSendRequests, while also preventing xrepl tablets deletion due to multiple calls to the function. {{<issue 22618>}},{{<issue 22608>}}
• Allows GarbageCollector subclasses to compile without declaring a virtual destructor. {{<issue 17598>}}

yugabyted

• Allows a smooth restart of the second node in a cluster using the join flag without throwing any errors. {{<issue 20684>}}

Bug fixes

YSQL

• Fixes YSQL upgrade single connection mode error preventing new connection attempts before the old ones are released. {{<issue 22283>}}
• Safeguards against an out-of-bounds memory write error in HashAggregate plan node with duplicate columns by rectifying array size allocation. Imported tests ensure the fix's robustness. {{<issue 21122>}}
• Corrects an issue where certain unbatchable filters weren't detected during indexpath formation when indexpath accepted batched values from multiple relations. {{<issue 21292>}}
• Resolves colocation option issues in table creation linked to table rewriting and partitioning. Enhances the defGetBoolean function to parse string values "0" and "1" as false and true respectively, and shifts a verification step to a earlier spot in the CREATE TABLE execution path, ensuring successful table partition creation. {{<issue 20302>}},{{<issue 20914>}}
• Corrects checks in YbIsScanCompatible to ensure the right-hand side (RHS) of all bound index conditions, not just inequalities, fits into the left-hand side (LHS) datatype. {{<issue 21758>}}
• Refines the YbGetOrdinaryColumnsNeedingPgRecheck condition to align with the ybIsTupMismatch implementation, ensuring Postgres rechecks index conditions when the "preliminary check" is skipped due to an invalid target key attnum. {{<issue 21451>}}
• Resolves compile errors in 2.18 build associated with MemTracker and reinstates the GetDbBlockCacheNumShardBits definition. {{<issue 21955>}}
• Ensures pushed down RowCompareExpressions correctly enforce non-null column references, rectifying previous behavior and enhancing data accuracy. {{<issue 21847>}}
• Stops Batched Nest Loop (BNL) crashes by ensuring better indexing condition checks. {{<issue 21954>}}
• Refines the logic to accurately push down join clauses to batched index scans without causing conflicts. {{<issue 21878>}}
• Grants BNL hashtable its own expression context to prevent data overwrites during query execution. {{<issue 21266>}}
• Allows yb_get_range_split_clause to handle GinNull decoding failure in GIN Index partitions without blocking backups. {{<issue 22356>}}
• Prevents coredumps by ensuring YSQL webserver destruction upon receiving a termination signal. {{<issue 18948>}}
• Prevents the IN expressions on single column from wrongly taking the tuple path, ensuring correct data processing. {{<issue 22704>}}

YCQL

• Allows the deletion of the Cassandra role in YCQLsh without it regenerating upon cluster restart, by adding a flag to mark if the role was previously created. {{<issue 21057>}}

DocDB

• Now ensures tablets are cleared from the pending deletes state even on task failures, preventing load balancer operations issues. Additionally, fixes a race condition to prevent indefinite retention of tablets in the set after task completion. {{<issue 13156>}}
• Fix ensures heartbeat processing doesn't blindly overwrite tablet replica state, avoiding potential data corruption. {{<issue 21836>}}
• Removes unnecessary flush during snapshot deletion, preventing write blocks. {{<issue 22369>}}
• Ensures Create Table operation fails if Alter Replication encounters an error, enhancing the reliability of replication setup. {{<issue 21732>}}
• Modifies memory consumption calculations for pending operations to ensure accurate rejection of new writes at bootstrap, preventing loading failures. {{<issue 21254>}}
• Fixes a segmentation fault in yb-master by checking for a null pointer before dereferencing it, addressing an issue in the CDC run on 2.23.0.0-b37-arm. {{<issue 21648>}}
• Reduces unnecessary logging during checkpoint operations by lowering INFO level logs to DEBUG_LEVEL, enhancing log readability. {{<issue 21658>}}
• Enhances load balancer efficiency by refining validation logic to block tablet replica additions only for those with a pending delete in progress on the same server, avoiding potential slowdowns during mass tablet replica moves. {{<issue 21806>}}
• Reduces disruptions by throttling the master process log messages related to "tablet server has a pending delete" into 20-second intervals. {{<issue 19331>}}
• Fixes hidden split parent tablets wrongly appearing as leaderless in the master's leaderless tablet endpoint, ensuring accurate load balance status and preventing potential issues with Point-in-Time Recovery (PITR) operations. {{<issue 21371>}}

CDC

• Prevents failures in decoding change events by refreshing cached_schema_details when executing a new GetChanges request if the client indicates a necessity for the schema. {{<issue 20698>}}

v2.18.7.0 - April 1, 2024 {#v2.18.7.0}

Build: 2.18.7.0-b30

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.7.0-b30

Improvements

YSQL

• Treats REFRESH MATERIALIZED VIEW as a non-disruptive change, preventing unnecessary transaction terminations. The default option, REFRESH MATERIALIZED VIEW NONCONCURRENTLY, modifies metadata but without making a disruptive alteration. {{<issue 20420>}}

Bug fixes

YSQL

• Ensures the Linux PDEATH_SIG mechanism signals child processes of their parent process's exit, by correctly configuring all PG backends immediately after their fork from the postmaster process. {{<issue 20396>}}
• Fixes table rewrite issue on non-colocated tables/matviews in colocated DB, ensuring the new table uses the original table's colocation setting. Includes a workaround for GH issue 20914. {{<issue 20856>}}
• Prevents simultaneous send of read and write operations in the same RPC request that could lead to inconsistent read results, by ensuring that, in case of multiple operations, all buffered ones are flushed first. {{<issue 20864>}}
• Increases the schema version of the default partition whenever you create a new partition, preventing erroneous data insertion into the default partition due to cache refresh issues. {{<issue 17942>}}
• Adjusts heartbeat mechanism to shut down when an "Unknown Session" error occurs, reducing log alerts. This benefits idle connections with expired sessions. {{<issue 21264>}}
• Allows BNL's on outer and inner tables, even if the inner table has "unbatchable" join restrictions that can't accept batches of inputs, enhancing queries with complex join conditions. {{<issue 21366>}}
• Corrects error in creating an index using both SPLIT AT and INCLUDE by properly considering included columns in computing split rows. {{<issue 11880>}}

DocDB

• Corrects the RPATH setting for certain OpenLDAP libraries, ensuring correct versions are used and eliminating potential system conflicts. {{<issue 21236>}}
• Corrects block cache metrics discrepancy by ensuring Statistics object passes into LRUCache from TableCache for accurate updates. {{<issue 21407>}}
• Resolves issues in the under-replicated endpoint algorithm, ensuring correct counting of replicas only when the block's minimum number of replicas has not been fulfilled yet, hence offering accurate replica tally for placement blocks. {{<issue 20657>}}
• Validates the use of two arguments for disable_tablet_splitting, addressing a previous condition where only one was required, thereby enhancing backup process reliability. {{<issue 8744>}}
• Updates the condition for HT lease reporting to ensure accurate leaderless tablet detection in RF-1 setup, preventing false alarms. {{<issue 20919>}}

CDC

• Reduces tablet split data loss risk by updating cdc_sdk_safe_time only when a GetChanges explicit checkpoint acknowledgment is received, improving data reliability. {{<issue 15718>}}
• Fixed the decoding of NUMERIC value in CDC records to prevent precision loss by ensuring that the decoded string is not converted to scientific notation if its length is more than 20 characters. Additionally, the fix involves using the string representation with no limit on length and employing the Postgres numeric_out method for decoding, which is identical to the decoding of numerics in a PG query. {{<issue 20414>}}

v2.18.6.0 - February 6, 2024 {#v2.18.6.0}

{{< warning title="Use 2.18.7.0 or later.">}} {{< /warning >}}

Improvements

YSQL

• Issues a notice for unsafe ALTER TABLE operations, including for ADD COLUMN...DEFAULT, to indicate existing rows won't be backfilled with the default value, enhancing user awareness. Suppression possible by setting ysql_suppress_unsafe_alter_notice gflag to true. {{<issue 19360>}}
• Added sorting capabilities to BatchNestedLoopJoin to return the rows in the same order as NestedLoopJoin {{<issue 19589>}}
• Replaces the ysql_max_read_restart_attempts and ysql_max_write_restart_attempts gflags with yb_max_query_layer_retries, applies limit to Read Committed isolation statement retries, and adjusts retry_backoff_multiplier and retry_min_backoff defaults. {{<issue 20359>}}

DocDB

• Allows tracing of UpdateConsensus API by enabling the collect_update_consensus_traces GFlag, providing trace logs for UpdateConsensus from remote followers while also ignoring request.trace_requested if both leader and follower doesn't support this change. {{<issue 19417>}}
• Allows customizing retryable request timeouts to respect client-side YCQL and YSQL timings, optimizing log replay and preventing the tserver from rejecting requests that exceed durations. Adjusts default retryable request timeout to 660 seconds and offers a configuration to eliminate server-side retention of retryable requests with FLAGS_retryable_request_timeout_secs =0. {{<issue 18736>}}
• Logs all instances of tablet metadata creation/updation, enabling additional insights for troubleshooting in cases of multiple meta records for the same tablet. {{<issue 20042>}}
• Automates recovery of index tables impacted by a bug, preventing performance degradation and disk size leak, by ensuring schema.table_properties.retain_delete_markers is reset to false when index backfilling is done. {{<issue 19731>}}
• Offers redesigned server level aggregation for metrics, thus introducing more metrics for enhanced debugging. Removes several unused URL parameters and makes the new output compatible with YugabyteDB Anywhere and YugabyteDB Aeon, preventing double-counting issues in charts. Drops unused JSON and Prometheus callbacks from MetricEntity for a cleaner design. {{<issue 18078>}}
• Enhances debugging by adding verbose logging for load balancer global and per-table state changes and including global state information when printing sorted load. {{<issue 20289>}}
• Reduces server initialization time by eliminating the accumulation of deleted tablet superblocks during startup, through a modification in the DeleteTablet operation. {{<issue 19840>}}
• Enables automatic recovery of index tables affected by a bug, verifying their backfilling status and correcting the retain_delete_markers property to enhance performance. {{<issue 20247>}}
• Integrates hard and soft server memory limit metrics into Prometheus, enabling better visualization of memory allocation and usage data, specifically in non-default value universes. {{<issue 20578>}}
• Enables control over the batching of non-deferred indexes during backfill via a new flag, improving index management. {{<issue 20213>}}

Bug fixes

YSQL

• Eliminates the issue where killing a background worker consumed a Proc struct without returning it, effectively limiting webserver restarts to eight times. {{<issue 20154>}}
• Introduces enhanced logging in YSQL webserver for better debugging of failures, by adding two new flags that log endpoint access and print basic tcmalloc stats respectively. The flags can be updated by sending a SIGHUP signal. Additionally, offers more visibility into the webserver's memory usage through the newly created :13000/memz and :13000/webserver-heap-prof. {{<issue 20157>}}
• Rectifies a segmentation fault issue in the Postmaster process in rare lock acquisition cases. {{<issue 20166>}}
• Enables postmaster restart when a backend process is unexpectedly terminated while in a critical section, preventing potential data corruption and system hangs. {{<issue 20255>}}
• Establishes :13000/memz and :13000/webserver-heap-prof endpoints to improve visibility into the webserver memory usage while managing tcmalloc stats and current allocations. Integrates a new runtime variable yb_pg_metrics.webserver_profiler_sample_freq_bytes to control tcmalloc sampling frequency. {{<issue 20157>}}
• Introduces the pg_stat_statements.yb_qtext_size_limit flag to control the maximum file size read into memory, preventing system crashes due to oversized or corrupt qtext files. {{<issue 20211>}}
• Limits the number of attempts to read inconsistent backend entries to 1000 and logs every hundredth attempt to prevent potentially infinite waiting loop. {{<issue 20274>}}
• Fixes a segmentation fault during cleanup phase in the webserver by ensuring MyLatch is used consistently throughout the process lifecycle. {{<issue 20309>}}
• Reduces excessive memory consumption during secondary index scans {{<issue 20275>}}
• Refines the computation of required batched/un-batched path parameters, eliminating the need to manually track non-batchable relations and simplifying logic. Fixes a bug affecting Nested Loop joins. {{<issue 19642>}},{{<issue 19946>}}
• Upgrades "Unknown session" error to FATAL permitting drivers to instantly terminate stale connections, minimizing manual user intervention. {{<issue 16445>}}
• Rectifies correctness issue when nested correlated subqueries uses sequential scan {{<issue 19694>}}
• Rectifies correctness issues when BatchNestedLoop join is used and the join condition contained a mix of equality and non-equality filters. {{<issue 20531>}}
• Rectifies correctness issue when join on inequality condition and join columns contains NULL values {{<issue 20642>}}
• Rectifies correctness issue when queries involving outer joins and aggregate uses BNL. {{<issue 20660>}}
• Corrects the Batch Nested Loop's optimization logic for proper handling of cases where the given limit matches the outer table's exact size, ensuring accurate query results. {{<issue 20707>}}
• Addresses a bug that caused backup failure due to the absence of yb_catalog_version, by ensuring the function's existence post-normal migration. {{<issue 18507>}}

DocDB

• Mitigates the issue of uneven tablet partitions and multiple pollers writing to the same consumer tablet by only applying intents on the consumer that match the producer tablet's key range. If some keys/values are filtered out from a batch, it will not delete the consumer's intents, as they may be needed by subsequent applications. Also guarantees idempotency, even if some apply records stutter and fetch older changes. {{<issue 19728>}}
• Reduces chances of transaction conflicts upon promotion by delaying the sending of UpdateTransactionStatusLocation RPCs until after the first PROMOTED heartbeat response is received, enhancing transaction consistency and accuracy. {{<issue 17319>}}
• Resolves potential WriteQuery leak issue in CQL workloads, ensuring proper execution and destruction of queries, while preventing possible tablet shutdown blockages during conflict resolution failure. {{<issue 19919>}}
• Unblocks single shard waiters once a blocking sub-transaction rolls back, by applying identical conflict check logic for both distributed transactions and single shard transactions. {{<issue 20113>}}
• Eliminates a race condition that can occur when simultaneous calls to SendAbortToOldStatusTabletIfNeeded try to send the abort RPC, thus preventing avoidable FATALs for failed geo promotions. {{<issue 17113>}}
• Deprecates the enable_process_lifetime_heap_sampling flag and allows to control tcmalloc sampling via the profiler_sample_freq_bytes flag alone, enhancing user control over database sampling settings. {{<issue 20236>}}
• Resolves instances of the leaderless tablet endpoint incorrectly reporting a tablet as leaderless post-leader change, by tweaking the detection logic to depend on the last occurrence of a valid leader, ensuring more accurate tablet reporting. {{<issue 20124>}}
• Allows early termination of ReadCommitted transactions with a kConflict error, enhancing overall system throughput by eliminating unnecessary blockages without waiting for the next RPC restart. {{<issue 20329>}}
• Modifies SysCatalog tablet's retryable request retention duration to consider both YQL and YSQL client timeouts, reducing the likelihood of request is too old errors during YSql DDLs. {{<issue 20330>}}
• Fixes FATAL errors occurring during tablet participant shutdown due to in-progress RPCs by ensuring rpcs_.Shutdown is invoked after all status resolvers have been shut down. {{<issue 19823>}}
• Handles backfill responses gracefully even when they overlap across multiple operations, reducing risks of crashes and errors due to network delays or slow masters. {{<issue 20510>}}

CDC

• Corrects the computation of the cdcsdk_sent_lag metric to prevent steep, disproportionate increases by updating the last_sent_record_time when a SafePoint record is spotted, in addition to DMLs and READ ops. {{<issue 15415>}}
• Adjusts table drop process to shift CDCSDK streams from ACTIVE to DELETING_METADATA state, enhancing the ability to add new tables to streams even after a table drop. {{<issue 20428>}}

Other

• Adjusts tserver start and tserver stop scripts to successfully terminate all running PG processes, irrespective of their PID digit count. {{<issue 19817>}}

v2.18.5.2 - January 17, 2024 {#v2.18.5.2}

{{< warning title="Use 2.18.7.0 or later.">}} {{< /warning >}}

Bug fixes

YSQL

• Fix BNL local join lookup equality function. {{<issue 20531>}}

CDC

• Fix addition of new tables to stream metadata after drop table. {{<issue 20428>}}

v2.18.5.1 - January 10, 2024 {#v2.18.5.1}

{{< warning title="Use 2.18.7.0 or later.">}} {{< /warning >}}

Improvements

YSQL

• In Read Committed Isolation, limit the number of retry attempts when the aborted query is retried. {{<issue 20359>}}

Bug fixes

YSQL

• Return correct results when Batch Nested Loop join is used for queries involving Nested LEFT JOINs on LATERAL views. {{<issue 19642>}},{{<issue 19946>}}

DocDB

• In Read Committed Isolation, immediately abort transactions when conflict is detected. {{<issue 20329>}}

v2.18.5.0 - December 21, 2023 {#v2.18.5.0}

{{< warning title="Use 2.18.7.0 or later.">}} {{< /warning >}}

New features

CDC

• Introduces a flag: enable_tablet_split_of_cdcsdk_streamed_tables (default:false) to toggle automatic tablet splitting for tables in a CDCSDK stream, enhancing user control over replication processes. {{<issue 19482>}}

Improvements

YSQL

• Enables the display of unique DIST statistics through the auto_explain module, available with a new option auto_explain.log_dist and can be turned off as desired. {{<issue 17508>}}
• Enables the modification of is_single_row_txn for finer control over non-transactional writes required by COPY, index backfill, or when yb_disable_transactional_writes is set, preventing issues during non-bufferable operations for single row transactions. {{<issue 4906>}}
• Modifies YSQLDump to generate create index NONCONCURRENTLY, preventing automated index backfilling during the backup-restore process, making it quicker and more effective. {{<issue 19457>}}
• Verifies error return of px_cipher_decrypt in pgcrypto, a crucial step towards supporting OpenSSL 3.0+. {{<issue 19732>}}
• Enables a prerequisite import of upstream PG commit Disable OpenSSL EVP digest padding in pgcrypto, paving the way for OpenSSL 3.0+ upgrade. {{<issue 19733>}}
• Allows for the import of upstream PG commit Add alternative output for OpenSSL 3 without legacy loaded, aiding OpenSSL 3.0+ upgrade. {{<issue 19734>}}
• Prevents potential SQL injections in extension scripts by rejecting substitutions of extension schemas or owners, mitigating vulnerability CVE-2023-39417. {{<issue 14419>}}
• Mitigates potential security issues by importing upstream Postgres commit from REL_11_STABLE for future support of ALTER <object> DEPENDS ON EXTENSION, although it's not currently available in YugabyteDB. {{<issue 14419>}}
• Refines the planner to recognize YB LSM indexes' order sustaining properties, even with IN conditions on non-leading columns, thus supporting ordered distinct index scans effectively. {{<issue 19576>}}
• Increase the oom_score_adj (Out of Memory score adjustment) of the YSQL webserver to 900, the same as PG backends, prioritizing its termination when it significantly consumes memory. {{<issue 20028>}}
• Enhances debugging capabilities in YSQL Webserver process by introducing two new flags for endpoint access logging and tcmalloc stats printing. {{<issue 20157>}}
• Enhances YSQL webserver with new /memz and /webserver-heap-prof endpoints for greater visibility into webserver memory usage. Also introduces a new runtime variable yb_pg_metrics.webserver_profiler_sample_freq_bytes for controlling tcmalloc sampling frequency. {{<issue 20157>}}

DocDB

• Prevents tserver heartbeats to a master leader in a different universe, preventing incorrect registrations and data loss risks, gated by a new autoflag master_enable_universe_uuid_heartbeat_check = true, requiring user to manually update post-upgrade. {{<issue 17904>}}
• Allows customization of MemTracker's metric names for better distinguishability, and enables aggregation of metrics at table-level for more comprehensive analysis. {{<issue 18627>}}
• Adopts the trace outside the block for ensuring correct execution of per-session tracing with standalone traces, and fixes callbacks to adopt the appropriate trace. {{<issue 19099>}}
• Prevents potential lags and errors in the WaitOnConflict workload testing by ensuring in-memory locks get properly resettled, even in the event of a failed conflict resolution sequence. This fix helps avoid lock-related issues that might occur with any leaked or stuck WriteQuery objects. {{<issue 18770>}}
• Introduces support for rocksdb_check_sst_file_tail_for_zeros flag, enabling verification of SST data file trailing zeros and reporting errors for anomalies detected. {{<issue 19691>}}
• Includes the version info in the error message for easier identification when the Yugabyte process incorrectly starts on an older version after AutoFlags enabled. {{<issue 16181>}}
• Adjusts the verbose level for VLOG macros to help differentiate between INFO and VERBOSE logs, fostering ease in debugging and analysis with better log filtration. {{<issue 15273>}}
• Adds a metric for the number of running tablet peers on a tserver, making it easier to compute the ratio of tablet peers to cores and memory on YBM clusters. {{<issue 9647>}}
• Allows for additional debug logs that offer relevant information for investigating instances of read path failure due to the expectation of a hybrid time error. {{<issue 19876>}}
• Rejects ConfigChange requests for system catalog while another server is transitioning, preventing potential data loss from mistaken quorum formation by new peers. {{<issue 18335>}}
• Enables SST write retries on detected corruption after write with the rocksdb_max_sst_write_retries flag, enhancing data reliability. {{<issue 19730>}}
• Replaces glog includes with yb/util, introducing yb VLOG macros for clearer differentiation between INFO and VERBOSE logs, while addressing issues of duplicate includes. {{<issue 15273>}}
• Balances tablet distribution across drives to prevent bottlenecks and better utilizes disk bandwidth during remote-bootstrapping of multiple tablets. {{<issue 19846>}}
• Upgrades OpenSSL to version 3.0.8 as the previous version reached End of Life, pulling in changes to disable Linuxbrew builds and update glog for stack unwinding based on the backtrace function. {{<issue 19736>}}
• Ensures the master_join_existing_universe flag value check before restoring the initial sys catalog snapshot to avoid incorrect operations. {{<issue 19357>}}
• Enables automatic recovery of index tables for bugs detected in the context of all new index tables created from 2.14; prevents retention of tombstones in SST files, mitigates performance degradation and disk size leak. {{<issue 19731>}}

yugabyted

• Corrects the CPU usage reporting in the sankey diagram by filtering nodes based on region selection on the performance page. {{<issue 19991>}}
• Adjusts the yugabyted start command to interpret 0.0.0.0 as 127.0.0.1 in the advertise_address, aligning with the IP use in master, tserver, and yugabyted-UI. {{<issue 18580>}}

Bug fixes

YSQL

• Allows ALTER TABLE DROP CONSTRAINT IF EXISTS command to execute without throwing an error, even when the table lacks a constraint, enhancing the user experience. {{<issue 17659>}}
• Trims down the probability of inaccurate behaviour involving conflicts between single shard INSERT operations by ensuring read times are chosen after conflict resolution, enhancing data consistency. {{<issue 19407>}}
• Addresses mismatches that occurred due to non-execution of the postprocess script on alternate expected files and default_expectfile in pg_regress. {{<issue 19737>}}
• Resolves a bug that hindered consecutive lock acquisition due to not releasing LWLocks during AbortTransaction, preventing PostgreSQL processes from getting stuck. {{<issue 19509>}}
• Introduces a per-database Postgres new object identifier (OID) allocation option, enabling unique OID in each database and supporting increased scalability for multi-node or multi-tenant environments. New config flag ysql_enable_pg_per_database_oid_allocator added to toggle between old and new allocator behavior. {{<issue 16130>}}
• Prevents potential postmaster crashes during cleanup of killed connections by using the killed process's ProcStruct to wait on an unavailable LWLock. {{<issue 18000>}}
• Prevents missed rows during distinct queries to tables with dead tuples by allowing AdvanceToNextRow to return false without seeking further if it encounters a dead tuple. {{<issue 19911>}}
• Corrects a bug affecting type-checking for bound tuple IN conditions involving binary columns like UUID, appearing in releases 2.17.1 and up. {{<issue 19753>}}
• Stabilizes the PgCatalogVersionTest.RecyleManyDatabases unit test by adding a delay in the ExternalMiniCluster::Restart routine to ensure full readiness of master/tserver, preventing potential deadlock situations. {{<issue 17849>}}
• Reduces spinlock deadlock detection time by 75% for prompt handling of potential freezes and restarts Postmaster when a process holding a spinlock is killed, ensuring successful initiation of new connections. {{<issue 18272>}},{{<issue 18265>}}
• Restarts postmaster for easier and more effective cleanup when a process is accidentally killed during its own initialization or cleanup, avoiding potential issues with shared memory items handling. {{<issue 19945>}}
• Adjusts the batched IN condition derivation in Batched Nested Loop queries to ensure accurate array type matching, preventing unexpected database crashes during query with aggregation and join operations. {{<issue 20003>}}
• Refines background worker processes structure to prevent loss of Proc struct after killing a background worker, ensuring continued webserver functionality. {{<issue 20154>}}
• Rectifies gcc11 compilation failure on CentOS by modifying iterator to non-const. {{<issue 20087>}}
• Resolves a segmentation fault issue in postmaster by enabling it to use the Proc struct of a killed process, thereby enhancing process cleanup during unexpected terminations. {{<issue 20166>}}
• Restart the postmaster whenever a process is killed while writing in a critical section, preventing corrupted state and CPU consumption due to infinite loops. {{<issue 20255>}}
• Caps the number of attempts to read inconsistent back-end entries at 1000 to prevent indefinite waiting, proceeding with potentially incomplete yet safe results if consistently fails. {{<issue 20274>}}
• Resolves segmentation faults in the webserver SIGHUP handler during cleanup. {{<issue 20309>}}
• Corrects an error during table rewrite processes that previously omitted the relforcerowsecurity, preserving the Row-Level Security (RLS) behavior. {{<issue 19815>}}

DocDB

• Overcomes issues with YCQL, snapshots, and metrics by migrating the TableInfo namespace_name field through the master migration framework, first introduced in version 2.3, with the namespace loader now preceding the table loader. {{<issue 17713>}}
• Allows setting all columns of a row to NULL, resulting in deletion instead of creating a row consisting of NULLs, rectifying an issue during compaction. {{<issue 18157>}}
• Allows tracing of outgoing calls only if the current RPC is being traced, reducing excessive memory consumption and logging. {{<issue 19497>}}
• Allows loading transactions data when tablets are opened after a split or restart without relying on the assumption of seeing a strong intent and avoiding process failures in DEBUG builds. {{<issue 18615>}}
• Restores the original behavior that doesn't count tablets on dead tservers towards the replica count in the under-replicated endpoint after a master leader failover. {{<issue 19514>}}
• Facilitates replay of alter schema during local bootstrap and resolves inconsistencies between checkpoint and metadata, enhancing reliability of the database operations. {{<issue 19546>}}
• Streamlines transaction processing by updating TabletState only for tablets engaged in writes and ignoring old statuses during transaction promotion, reducing failure errors and boosting consistency. {{<issue 18081>}},{{<issue 19535>}}
• Prevents premature garbage collection of schema packings on the target cluster in xCluster configuration, ensuring that schema versions referenced will not be removed before any related rows are received, enhancing the stability of your databases during schema changes or network partitions. {{<issue 17229>}}
• Reduces index size growth and enhances query speed by properly updating the retain_delete_markers field on index backfilling, addressing an issue where delete markers persisted in SST files. {{<issue 19544>}}
• Refines meta cache updates to avoid overwriting child tablets and consequently causing stale data, ensuring more accurate partition map refreshes. {{<issue 18732>}}
• Increases tablet_creation_timeout_ms from 30 seconds to 600 seconds to avoid tablet replacement during table creation, enhancing restore operations' success rate under high network latency conditions. {{<issue 13030>}}
• Resolves a use-after-free issue in RefinedStream's failure path to enhance system stability and prevent memory corruption. {{<issue 19727>}}
• Synchronizes Master and TServer's MemTable metric names to improve monitoring consistency, aiding in the visibility of total memory consumption across all tablets in the Master's memory breakdown UI page. {{<issue 19904>}}
• Fixes an intermittent index creation failure by checking the result of is_running, ensuring stability for empty YCQL tables, and preventing retain_delete_markers from being incorrectly set. {{<issue 19933>}}
• Resolves a concurrency issue in MasterChangeConfigTest by operating on a copy of ExternalMaster* to prevent timing out when removing a follower with a transitioning server. {{<issue 19927>}}
• Allows a single heap profile run at a time, preventing concurrent running that causes inconsistent sampling frequencies, enhancing the accuracy of these profiles. {{<issue 19841>}}
• Resolves data inconsistency between source and target instances during xCluster +Transactions +Aggressive tablet Splitting operations. Now, ensures accurate application of intents matching the producer tablet's key range on the consumer, facilitating correct data replication in diverse transaction scenarios. {{<issue 19728>}}

CDC

• Allows continuation of tablet fetching, even if certain tables face errors, by logging a warning instead of sending unnecessary errors to the client. {{<issue 19434>}}
• Returns error to the client if reading the cdc_state fails, ensuring consistent and correct list of tablets, enabling your client to take suitable action. {{<issue 19428>}}
• Eliminates deadlocks during database drops by properly handling namespace-level CDC stream deletion and setting up the database deletion process to finish in a finite time. {{<issue 19879>}}
• Adjusts CDCSDK's Catalog Manager to delete entries from cdc_state on new tablets, resolving race conditions due to tablet split and ensuring accurate metadata cleanup in cdc_state table for dropped tables. {{<issue 19746>}}

Other

• Avoids a segmentation fault in the stats collector following a Postmaster reset, solving potential crashes experienced when queries are terminated by SIGSEGV or SIGKILL. {{<issue 19572>}}

• Eliminates potential crashes by adding nullity checks to CDCServiceImpl::GetChanges that previously could return a null pointer. {{<issue 19524>}}

• Allows for the dynamic adjustment of previously hardcoded ports such as master_rpc_port, tserver_webserver_port, and master_webserver_port based on user's custom configurations, thereby facilitating successful creation of multi-region/zone clusters even with non-default port settings. {{<issue 15334>}}

v2.18.4.2 - November 8, 2023 {#v2.18.4.2}

Build: 2.18.4.2-b2

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.4.2-b2

This is a YugabyteDB Anywhere-only release, with no changes to the database.

v2.18.4.1 - November 6, 2023 {#v2.18.4.1}

Build: 2.18.4.1-b3

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.4.1-b3

Bug fix

• [18081] [19535] [DocDB] Update TabletState only for tablets that are involved in write, Ignore statuses from old status tablet on promotion

v2.18.4.0 - October 25, 2023 {#v2.18.4.0}

Build: 2.18.4.0-b52

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.4.0-b52

Highlights

• The yugabyted utility for deploying and managing clusters is now GA for production deployments.

New features

• [18748] [CDCSDK] Change default checkpointing type to EXPLICIT while stream creating

Improvements

• [12751] [CDCSDK] Made maxAttempts and sleepTime for retrying RPCs configurable in AsyncClient
• [16780] [YCQLSH] Updating yugabyte-db-thirdparty release version
• [17270] [DST] Fast PITR: Add ability to set cotables filter map
• [17360] [DocDB] Dynamically set the db_block_cache_num_shard_bits based on the number of cores
• [17464] [DST] Fast PITR - Use cotables filter for master restore
• [17716] [DocDB] Move tcmalloc utility code out of MemTracker
• [17758] [18561] [DocDB] Enable allocation sampling endpoint for gperftools tcmalloc.
• [18207] [DST] Fast PITR - Custom history retention for DocDB metadata table from other tables of sys catalog tablet
• [18482] [yugabyted] Improve logging of yugabyted-ui, and changes to make it more robust against node failures
• [18668] [DocDB] Block stack collection during thread creation
• [18698] [xCluster] Check for existence of data in table for IsBootstrapRequired check of a new stream
• [18818] [yugabyted] Cloud based fault tolerance is supported with the yugabyted start command.
• [18998] [DocDB] New flag to force shell mode for empty masters regardless of master_addresses.
• [19071] [DocDB] Add estimated bytes / count to pprof memory pages
• [19097] [DocDB] Escape output of master and tserver UI endpoints
• [19176] [DocDB] Add URL param to disable Prometheus metrics help and type metadata
• [19196] [CDCSDK] Add test case for tablet split followed by Drop table
• [19203] [xCluster] Add xCluster tserver UI page
• [19221] [DocDB] Improved timeout handling for YCQL index scan
• [19244] Update logo in yugabyted UI
• [19454] [Build] Move macOS support off of mac11
• [19482] [CDCSDK] Add GFlag to disable tablet split on tables part of CDCSDK stream
• [19524] [xCluster] Check for nullptr of CDCTabletMetrics

Bug fixes

• [13128] [YSQL] Fix ClearSeparateDdlTxnMode error handling
• [13999] [DocDB] Implement code and tests for /api/v1/tables API, /api/v1/table?id=X JSON API endpoints
• [14658] [yugabyted] Deleting the conf directory when yugabyted destroy command is used.
• [17025] [xCluster] Calculate replication lag metrics for split tablet children
• [17727] [DocDB] Use allocated_size instead of size / count in tcmalloc sampling report.
• [17959] [yugabyted] Use new /table API on the tablets page, and new /api/v1/tablets tserver API in go apiserver. Migrate the yugabyted API server to use /api/v1/tables endpoint instead of HTML parsing
• [18379] [DocDB] Retain schema packing while creating snapshot
• [18388] [yugabyted] Fixing yugabyted start --secure command
• [18408] [yugabyted] Fixing /api/tables to divide tables and indexes
• [18482] [yugabyted] Fix total disk usage on overview page
• [18658] [xCluster] Skip local peer during bootstrap if not ready/valid
• [18738] [yugabyted] Corrected the spelling error in the cert command description when displaying the help output.
• [18744] [DocDB] Add ability to recover from follower lag caused by stuck OutboundCall
• [18744] [DocDB] Fixed issue with last_rpc_start_time_ reset which resulted in tsan race in connection
• [18770] [DocDB] Stricter memory order in PreparerImpl::Run to prevent re-ordering of instructions
• [19017] [yugabyted] yugabyted UI redirects should work with page refresh
• [19292] [CDCSDK] Publish snapshot_key in GetCheckpointResponse only if it is present
• [19323] [yugabyted] Fixes on the yugabyted UI
• [19348] [CDCSDK] Only delete removed tablets from cdc_state in CleanUpCDCStreamsMetadata
• [19384] [YSQL] Fix handling of case where a given RowCompareExpression cannot be bound
• [19385] [CDCSDK] Fix WAL GC issue for tables added after stream creation
• [19394] [CDCSDK] Only populate key to GetChangesRequest when it is not null
• [19394] [CDCSDK] Set snapshot key from correct parameter in explicit check pointing
• [19414] [yugabyted] UI bugs
• [19428] Return error to client if scan of cdc_state fails
• [19434] [CDCSDK] Do not completely fail while fetching tablets if one table hits error
• [19440] [DocDB] Fix bug where invalid filter key was passed to Iterator initialization in backwards scans

v2.18.3.1 - October 19, 2023 {#v2.18.3.1}

Build: 2.18.3.1-b1

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.3.1-b1

Bug fixes

• [19440] [DocDB] Fix bug where invalid filter key was passed to Iterator initialization in backward scans

v2.18.3.0 - September 20, 2023 {#v2.18.3.0}

Build: 2.18.3.0-b75

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.3.0-b75

New features

• [18069] [YSQL] Add support for OIDC (JWT) based authentication

Improvements

• [12853] [PITR] Turn consistent_restore flag on by default
• [16963] [#18838] [DOCDB] Add verbose logging to the load balancer
• [17570] [DOCDB] Improve leaderless tablet endpoint
• [18069] [YSQL] Introduce jwtcpp_util as a wrapper over JWT-CPP
• [18321] [YSQL] Enable preloading additional tables by specifying catalog list in flags; restore --ysql_catalog_preload_additional_tables
• [18465] [DOCDB] Dump PeerManager to tserver debug ui consensus page
• [18555] [CDCSDK] Refactored cdcsdk_ysql_tests to add waiting mechanism for GetChanges() call
• [18826] Build - thirdparty_tool --list-compilers option enhancement
• [18854] [YSQL] SELECT INTO and CREATE TABLE AS cause catalog version increase

Bug fixes

• [12574] [xCluster] Hold reference count of YBTable in XClusterWriteRpc
• [13171] [DOCDB] reject expired retryable requests
• [14210] [DOCDB] fixed YSQL tablet routing
• [14369] [DOCDB] Add a flag to ignore NULL sys catalog entries during enumeration
• [15925] Disabling pg_tablet_split-test.cc tests for tsan
• [16432] [17080] [xCluster] Disallow N:1 replication topology
• [16446] [YSQL] Make yb_pg_batch_detection_mechanism configurable at runtime for session, Peek message following Execute from JDBC driver to identify batch execution
• [16556] [CDCSDK] Only ignore unpolled tablets depending on context
• [17146] [xCluster] Fix missing pgschema_name during SetupUniverseReplication
• [17148] [DOCDB] Fixed wrong metric_type of prometheus metric
• [17468] Retry ysqlsh in the Docker-based package check
• [17682] [DOCDB] fix hybrid time too low due to race between TransactionParticipant::Poll and RaftConsensus::Start in rf-1 setup
• [17751] [CDCSDK] Turn flag enable_update_local_peer_min_index to true by default
• [17795] [CDCSDK] Don't stream same txn twice in same GetChanges call
• [18164] [DOCDB] Ignore deleted tablets in leaderless endpoint
• [18281] [YSQL] Fix query layer's read restart retries in Read Committed isolation
• [18344] [YCQL] Fix TOJSON() to also unpack nested FROZENs
• [18387] [DOCDB] Select statement fails with TABLET_DATA_SPLIT_COMPLETED
• [18518] [DOCDB] removed unused MemTracker::rand_ field
• [18519] [CDCSDK] Stop sending checkpoint details for unpolled child tablets in UpdatePeersAndMetrics
• [18586] [YCQL] Return reserved keywords in double quotes
• [18592] [DOCDB] Fix log message format in WritePostgresConfig
• [18635] [xCluster] Fix ordering in GetLatestEntryOpId during bootstrap
• [18642] [DOCDB] Code refactoring to match usage of has_been_fully_compacted to a proper name
• [18652] [YSQL] YB Sequential Scans don't take locks in SERIALIZABLE isolation
• [18879] [YSQL] Disable pushdown for TextSearch functions that are not thread safe

Known issues

N/A

v2.18.2.1 - August 16, 2023 {#v2.18.2.1}

Build: 2.18.2.1-b1

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.2.1-b1

New features

• [16912] [DocDB] Expose GetOldTransactions RPC to fetch long-running transaction metadata
• [17312] [CDCSDK] Add begin/commit records for single shard txns
• [17608] [DocDB] Add HTTP security headers
• [17696] [YSQL] Add yb_latency_histogram to server statements
• [17700] [YSQL] Add yb_get_percentile YSQL function
• [17795] [CDCSDK] Add wal_segment_index to GetChanges Response and Request
• [17868] [DocDB] Export histogram quantile p50/95/99 as gauge to Prometheus #TYPE
• [18068] [YSQL] add a new tserver (preview) convenience GFlag ysql_ident_conf_csv for user name maps

Improvements

• [14308] [DocDB] update flag description now that savepoints work with xCluster
• [15060] [DocDB] Makes backoff loop aggressive in colocated table creation path
• [16116] [DocDB] Lazily flush superblock on colocated table creation
• [16466] [DocDB] Flush superblock before reading in RBS with lazy superblock flush
• [16604] [DocDB] Renames last_change_metadata_op_id to last_flushed_change_metadata_op_id in superblock
• [16778] [DocDB] Background flush task when lazy superblock flush is enabled
• [17104] [DocDB] Shuts downs superblock_flush_bg_task_ on tserver shutdown
• [17356] [YSQL] Reduced memory_tracking overhead in high use situations from 10% to 3%
• [17683] [DocDB] Invalidate YCQL system.partitions cache on table creation
• [17730] [YSQL] Import Improve pruning of a default partition
• [17801] [YSQL] Option to toggle distinct pushdown
• [17805] [17965] [YSQL] Enhance TServer PG daemon robustness
• [17826] [17823] [YSQL] Use a more granular clock for RPC timing on Linux builds
• [17855] [CDCSDK] Force add snapshot bootstrap operation checkpoint to cdc_state table
• [17867] [DocDB] Account for table placement and read replicas in underreplicated endpoint
• [17884] [CDCSDK] Add checkpoint details to snapshot records and change check of copying checkpoint from colocated snapshot to streaming
• [17966] [DocDB] Unify ModifyPlacementInfo string parsing
• [17972] [CDCSDK] Add guards around updating invalid checkpoints during snapshot phase
• [18064] [DocDB] enhance tserver consensus state page
• [18067] [YSQL] Import 'Remove arbitrary restrictions on password length.'
• [18067] [YSQL] Import "Make StringInfo available to frontend code."
• [18069] [YSQL] Update third-party dependencies & add CMake configs for JWT-CPP
• [18116] [CDCSDK] Persist the safe_time from explicit checkpoint in cdc_state table
• [18391] [17864] [YSQL] RPC timers in write path default to granular clocks in Linux
• [18418] [CDCSDK] Set cdc_populate_safepoint_record default to true
• [DB-2613] [YSQL] Remove unused log4j reference from the java modules

Bug fixes

• [13395] [YSQL] Return error for unset bind PK value
• [14762] [DocDB] Prevent crashing the master if trying to restore a deleted table
• [16267] [17755] [DocDB] Fix issues in RBS from closest peer codepath
• [16441] [YSQL] Retry CREATE DATABASE when oid collision happens
• [16508] [YSQL] Fix incorrect tablet count assignment on user tables created within geo-partitoned tablespace
• [16597] [DocDB] Fixed PartitionSchema::CreateHashPartitions to provide even hash codes distribution
• [16665] [14308] [DocDB] fix logging of RocksDB write batches when packing is used, part I and II
• [16680] [CDCSDK] Fix colocated before image test with packed row
• [16711] [DocDB] Only display AutoFlags in UI that are relevent to the process
• [17038] [DocDB] Async open table while updating indexes
• [17043] [YSQL] Fix check to see when to attempt binding hash key options in scanspec
• [17225] [YSQL] Use consistent read snapshot in DEFERRABLE READ ONLY mode
• [17290] [DocDB] Fix race condition in deadlock_detector causing invalid probe_num
• [17351] [yugabyted] Enable EAR command is failing on Mac builds
• [17400] [17488] [16665] [14308] [DocDB] fix crash due to speculatively decoding a slice ending in a Typeset
• [17512] [yugabyted] Code changes to provide the master and tserver webserver port information to the API server
• [17558] [DocDB] Issue with picking local_limit during paging read
• [17574] [DocDB] Add EncryptedWritableFile::Size() to fix incorrect WAL footer start offset
• [17592] [17760] [DocDB] Rename WAL file after writing the file header on the disk
• [17648] [YSQL] Unexpected SELECT DISTINCT behaviour with non index conditions
• [17663] [DocDB] Clear drive load / leader load maps before appending to them
• [17718] [xCluster] Check table leader term in xCluster Poller
• [17795] [CDCSDK] Emit CDC records in order of commit time instead of op ID and fix response safe_time calculation in corner cases
• [17848] [DocDB] Fix race on RemoteTablet::replicas_ between RemoteTablet::GetRemoteTabletServers and RemoteTablet::Refresh
• [17869] [YSQL] fix incorrect optimization on NULL scans
• [17887] [DST] PITR restores should ignore Deleted namespaces
• [17890] [xCluster] create intent iterator outside of RegularDB batch writer to avoid deadlock
• [17925] [CDCSDK] Refetch correct schema in cases of schema versioning errors
• [17935] [CDCSDK] Reduce the number of txns in TestCDCSDKConsistentStreamWithManyTransactions
• [17961] [18008] [YSQL] Fix max connection limit errors with additional locking
• [17969] [CDCSDK] Fix heap-use-after-free failure for TestFromOpIdInGetChangesResponse
• [17981] [yugabyted] yugabyted start should work when custom certs_dir is provided
• [17997] [xCluster] IsBootstrapRequired should return False if any data exists in the WAL
• [18001] [DocDB] Skip key registry decryption in EncryptionManager::GetFullUniverseKeyRegistry() when EAR is disabled
• [18023] [DocDB] Log the result of chronyd when tserver crashes with a high clock skew
• [18071] [yugabyted] Changing the status display to not show the actual password when it's generated in secure mode
• [18133] [CDCSDK] GetTabletListToPollForCDC should return safe_time as well
• [18184] [DocDB] Fix crash in case of IN handling
• [18227] [DocDB] Fix log reader missing max_replicate_index
• [18322] [YSQL] fix index ScanKey bugs
• [18328] [CDCSDK] Fix bug in GetTabletIdsToPoll when only 1 child tablet has been polled
• [18393] [DocDB] Fix unknown int64 flags incorrectly getting tagged as runtime
• [18445] [CDCSDK] Call ResolveIntents if consistent_safe_time is stuck
• [18467] [CDCSDK] Do not send safepoint records in snapshot phase
• [18635] [xCluster] Fix ordering in GetLatestEntryOpId during bootstrap
• [18756] [xCluster] Only Throttle xCluster Traffic from source side

Known issues

N/A

v2.18.1.0 - June 27, 2023 {#v2.18.1.0}

Build: 2.18.1.0-b84

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.1.0-b84

Highlights

• Improvements to xCluster, including:
  • Support for transactional atomicity
  • Faster bootstrapping
  • Automatic transfer of the source universe certificate when new nodes are added to the target universe
  • Ability to set up replication even if matching tables do not exist on the target universe
  • Improved performance and reliability when gathering replication stream statuses
  • Disallowing replication between two YugabyteDB versions that are incompatible
• View latency histograms for queries using the yb_latency_histogram column in pg_stat_statements
• Faster nested loops with batching. Requests to YB-TServers are batched during joins using a new join execution method referenced as Batched Nested Loop, and derived from classic nested loop joins. {{<tags/feature/ea>}}

New features

• [14445] [YSQL] Add query latency histogram column in pg_stat_statements
• [15036] [xCluster] Add transactional flag to setup_universe_replication
• [16294] [14067] [YSQL] Implementation Enhancements for EXPLAIN ANALYZE
• [16757] [xCluster] Add transactional flag on Producer stream

Improvements

• [13786] [17395] [17494] [DOCDB] Auto full compactions based on deletes in SST files
• [14284] [13041] [YSQL] Avoid setting read time when using serializable isolation level
• [14473] [DOCDB] Accept registrations from tservers on the same node with greater sequence numbers
• [16240] [YSQL] Use per-tuple memory context in relcache preloading
• [16343] [DOCDB] Include TableType in the table and cdc metrics
• [16750] [xCluster] Disallow setting up transactional replication if any YCQL tables are included in replication
• [16826] [DOCDB] [xCluster] Use RocksDB DirectWriter for applying external intents
• [17013] [DOCDB] Add colocation info to ListTables output
• [17016] [DOCDB] Add #HELP and #TYPE lines for prometheus metric
• [17150] [YSQL] Allow BNL on expressions with multiple relations on one side
• [17318] [YSQL] Import 'Replace last PushOverrideSearchPath() call with set_config_option().' from PG
• [17358] [DOCDB] Add tracking for memory used by YBMetadataCache
• [17455] [xCluster] Log when UpdateCdcMetrics is delayed
• [17463] [xCluster] Producer and consumer side network throttling
• [17633] [YCQL] Introduce flag to control local calls on RPC thread

Bug fixes

• [14308] [xCluster] fix xCluster replication so it correctly handles SAVEPOINTs
• [14926] [DOCDB] Prevent intent clean-up while reading YSQL system tables
• [15769] [YSQL] Fix functional indexes returning wrong results
• [16157] [17433] [YSQL] Set PgClient session timeout value
• [16367] [DOCDB] Delete Snapshot keeps on retrying indefinitely after master failover
• [16513] [DOCDB] Correcting statistics for DOCDB keys found
• [16590] [DOCDB] Add test which ensures deadlocks are detected across partitions
• [16612] [DOCDB] Fix transaction promotion race condition resulting in unnecessary abort
• [16631] [DOCDB] Fix delete_snapshot sending delete RPCs indefinitely for deleted tablets
• [16697] [DOCDB] Disallow multiple OpenTable calls in YBMetaDataCache::GetTable
• [16698] [DOCDB] Add NewWritableFile Encryption Wrapper to fix WAL reuse issue
• [16794] [DOCDB] Make full compactions non-exclusive by default
• [16805] [DOCDB] Split PgMiniTest into smaller ones
• [16863] [DOCDB] Ignore missing YSQL indices when importing a snapshot
• [16911] [DOCDB] Augment GetLockStatus tserver endpoint to handle transaction_id list
• [16943] [xCluster] switch xcluster_ysql-test to use consistent WAL mode
• [16999] [DOCDB] Retry read if read buffer was full on previous step
• [17026] [xCluster] Fix metrics jump when regaining leadership
• [17041] [YCQL] Add cache in FindMemberForIndex for jsonb
• [17074] [DOCDB] Root memory consumption should not include pageheap_free_bytes
• [17107] [YCQL] Prevent DROP TYPE for the used YCQL type
• [17163] [YCQL] Use Metadata Cache by default in PREPARE
• [17194] [CQL] Add memtracker to track CQL requests memory usage
• [17203] Fix post_install.sh failure related to pg_stat_statements.so
• [17219] [YCQL] Fixing tests AlterDropAdd*
• [17253] [DOCDB] Fix RPC error handling in pg client
• [17261] [DST] [PITR] Single file compaction does not reset Hybrid Time filter
• [17276] [xCluster] cleanup the transaction to write_id map whenever transaction is applied
• [17293] [xCluster] Avoid creating multiple copies of GetChangesPB and cleanup PBs as soon as apply finishes
• [17296] [xCluster] Fix StreamMetadata TSAN race
• [17308] [DOCDB] [xCluster] track schema packings in frontier for external records
• [17317] [YSQL] Import Handle RLS dependencies in inlined set-returning functions properly
• [17362] Avoid double memory tracking of request data and fix tracking for plain response buffer size
• [17380] [YSQL] Make sure ybctid_column_value is still sent in index scans for upgrade compatibility
• [17398] [CQL] Account for user query and keyspace name for prepared statements
• [17404] Use LLVM URL from the third-party dir for ASAN/TSAN
• [17431] [DOCDB] Fix partial hash key columns projection in DocRowwiseIterator
• [17437] [DOCDB] Rename WAL reuse flag to include unit
• [17480] [DOCDB] Update to NULL could be lost after compaction when packed row is enabled
• [17502] [DOCDB] Exclude yugabyted-ui from sed binary manipulation
• [17502] Fix the shared library interpreter used for yugabyted-ui
• [17531] [DocDB] Disable wait-on-conflict behavior for YCQL traffic
• [17559] [DST] Adding loop call during tablet server init to wait on master leader to get encryption keys at rest
• [17632] [DOCDB] Fix GC of schema packings of colocated table
• [17712] [DOCDB] Fix system partitions query to not return empty namespace name when its not set at table level
• [17714] [YSQL] Resolve version_metadata.json missing issue
• [17726] [DOCDB] A workaround for memory leak from InboundCall weak pointers
• [17762] Bound the number of cloned ops by the number of partitions for IN conditions sent to hash sharded tables

Known issues

N/A

v2.18.0.1 - May 30, 2023 {#v2.18.0.1}

Build: 2.18.0.1-b4

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.0.1-b4

Bug fixes

• [17431] [DocDB] Fix partial hash key columns projection in DocRowwiseIterator
• [17480] [DocDB] Update to NULL could be lost after compaction when packed row is enabled
• [17502] [DocDB] Exclude yugabyted-ui from sed binary manipulation, Fix the shared library interpreter used for yugabyted-ui.

Known issues

N/A

v2.18.0.0 - May 16, 2023 {#v2.18.0.0}

Build: 2.18.0.0-b65

Third-party licenses: YugabyteDB, YugabyteDB Anywhere

Downloads

Docker

docker pull yugabytedb/yugabyte:2.18.0.0-b65

New features

• [12417] [YSQL] Use relcache init files in per-database catalog version mode
• [13336] [YSQL] push down the avg() aggregate
• [13488] [xCluster] Create index in transactionally consistent xCluster
• [14234] [14674] [DOCDB] Support Colocated+Packed+XCluster & auto pause/resume on schema changes
• [14665] [DOCDB] enable packed row for co-located tables by default
• [16068] [DOCDB] Support storing wait-for probes of distributed waiter txns spanning across tservers
• [16137] [DST] PITR + Sequences - Ensure generated sequence values are monotonic after restore
• [16165] [16358] [yugabyted] [UI] New design implementation for Overview and Database pages (#16167)
• [16165] [yugabyted] UI Adding replication factor info in the /cluster API
• [16195] [DOCDB] Add metrics support for ScanForward API
• [16218] [DOCDB] Adding xcluster_read_only mode to standby cluster databases
• [16351] [DocDB] Add Cql Tests to Simulate Update Index Callback Leak
• [16370] [DOCDB] enable packed row in locking unit tests
• [16380] Use thread safety analysis annotations to enforce the reactor thread requirement
• [16447] [DOCDB] Use shared memory for Perform queries between postgres and tserver
• [16503] [CDCSDK] Support colocated table with before image
• [16513] Add statistics to track DOCDB keys found (total, obsolete, and obsolete past history cutoff)
• [16515] [xCluster] Enable using metacache in cdcservice
• [16571] [DOCDB] Protect InboundCall::trace_ with the mutex
• [16600] Odyssey connection pooler CMake integration
• [16601] [DOCDB] Introduce simplified version for RocksDB iterator Next
• [16648] [DOCDB] Decode directly to QLValuePB
• [16663] [YSQL] Extend Batched Nested Loop Join functionality to Appended base relations for partitioned tables
• [16700] [DOCDB] Read directly to QLTableRow
• [16719] [YSQL] Make ysql_yb_bnl_batch_size a runtime flag
• [16742] Adding additional logging for API server
• [16746] [xCluster] Compute and propagate apply_safe_time from producer
• [16892] [DOCDB] AutoFlags should provide the capability to enable flags on Install scenarios only, but not upgrade

Improvements

• [3090] [YCQL] Limit topology change events to subscribed connections
• [13701] Build: Use Google TCMalloc for fastdebug build
• [14961] [14962] [YSQL] Further reduce first-query metadata RPCs
• [15014] [14333] [YSQL] Refactor some code to honour coding best pratices
• [15746] [DOCDB] harden master leaderless tablet endpoint
• [15749] [DOCDB] Delay rocksdb gauge metrics update to end of request
• [15794] [DOCDB] Reduced Number of Tablets in AreLeadersOnPreferredOnly test
• [15884] [YSQL] Skip indexed table's stats update after non-concurrent create index
• [15909] [DOCDB] Move sys catalog writes out of catalog loader
• [16058] [CDCSDK] Use safe time from request rather than response
• [16091] [DOCDB] Changing post-split compaction to run non-exclusively
• [16123] [DOCDB] Remove co-partitioning logic
• [16237] [xCluster] deprecate flags cdc_max_apply_batch_num_records and cdc_max_apply_batch_size_bytes
• [16255] [YSQL] Reduce memory use on insert, update, and delete by using per-tuple contexts and freeing bitmapsets
• [16317] [13718] Move common functionality to base iterator class
• [16374] [DOCDB] Improve error message when SAVEPOINT feature is attempted when disabled by --enable_pg_savepoint
• [16397] [DOCDB] Set table to RUNNING state only after all tablets are ready
• [16402] [yugabyted] Set cluster config as soon as the 3rd node joins
• [16442] [DST] PITR + Per DB catalog version: Increment only the catalog version of the relevant database
• [16454] [YSQL] Log on unexpected backend termination
• [16460] [YSQL] Improve error messages for transaction conflicts
• [16461] [YSQL] Allow hash batching to apply to an unbounded number of permutations
• [16467] compact_table Now Updates last_full_compaction_time For Tables without Writes
• [16472] [DOCDB] Replace std::thread with yb::Thread
• [16480] Reduce the amount of recompilation due to Postgres header copying
• [16494] [DOCDB] Create VerifyPgClientServiceCleanupQueue unit test
• [16504] [CDCSDK] [Colocation] YBTable object should contain metadata about colocation
• [16510] [xCluster] Return nullopt from GetSafeTime if namespace does not require xCluster safe time
• [16513] [DOCDB] Improve collecting statistics for DOCDB keys found
• [16545] Build: build/package on mac11
• [16551] [xCluster] Account for xcluster safe time for history retention on consumer
• [16555] [DOCDB] Avoid decoding doc hybrid time during read when possible
• [16562] [DOCDB] Avoid unnecessary calls to SeekForward and FindLatestValue in doc reader prepare phase
• [16581] [DOCDB] Remove outstanding tablet split limit by default
• [16587] [DOCDB] Move common ScanSpec functionality to base class and remove specific handling from iterator
• [16588] build-infra: Don't use pipes python module
• [16617] [DB-6011] [YCQLSH] Updating yugabyte-db-thirdparty release version
• [16650] [DOCDB] Remove intent_doc_ht control field from packed column value
• [16651] [DOCDB] Sanity check for RocksDB iterator status validation
• [16666] [YSQL] Switch to new query context for get_tablespace_distance() operations
• [16683] Support build with Clang 16
• [16715] [DOCDB] Optimize handling of add_table change_metadata_op
• [16756] [xCluster] Move yb-admin get_xcluster_estimated_data_loss to get_xcluster_safe_time
• [16789] [YSQL] Assume that CAPABILITY_PickReadTimeAtTabletServer is enabled all the time
• [16849] [DOCDB] Tablet splitting: Lower the thresholds for splits on low core machines
• [16868] [DOCDB] Sanitizes debug messages
• [16875] [DST] PITR - Extend master's sys catalog retention based on snapshot schedule
• [16900] [CDCSDK] Remove cdcsdk_unprocessed_tables from CDCStreamInfo
• [16969] yugabyted UI updates
• [16991] [YSQL] Update pruning quals when nested loop batching is on
• [16993] [YSQL] Restructure type checking when forming batched expressions
• [17075] [17078] Send full universe key registry to Tablet Server on Startup
• [17115] [YSQL] Restructure batched indexqual creation
• [CLOUDGA-12738] Make the package updates work on AlmaLinux 9
• [CLOUDGA-12738] Update OS pacakges during AMI build process
• [DB-5975] [YCQLSH] Updating yugabyte-db-thirdparty release version

Bug fixes

• [506] [YCQL] Disable alter table rename on columns with index
• [13862] [YSQL] Fix metric for total number of connections made
• [14167] [YSQL] Fix PgPackedRowTest.AddDropColumn flakiness
• [14270] [DOCDB] Fixes test SnapshotTxnTest.BankAccountsWithTimeStrobe
• [15090] [DOCDB] Tablet splitting: Some queries fail with "The tablet is shutting down" during the workload
• [15166] [DOCDB] Fix crashes due to tablet shutdown timeout
• [15449] [CDCSDK] Send snapshot end marker when snapshot has no data
• [15612] [YSQL] Fix a memory leak in pg_isolation_regress
• [15930] [DOCDB] Fixes data race issue with YBSubTransaction in TableSizeTest_PartitionedTableSize
• [16009] [YSQL] Fix the system catalog versioning behaviour on CREATE TABLE with FK reference command
• [16010] [DOCDB] Fixes tsan data race in CompactionTestWithFileExpiration.FileExpirationAfterExpiry
• [16131] [DST] B/R, PITR Fix invalid enum value of restoration state object
• [16156] [DOCDB] Limit duplicate address check to used host-port combination in Tserver registration
• [16195] [DOCDB] Handle duplicate user keys in ScanForward API
• [16262] [DOCDB] deflake failover pitr tests on tsan
• [16329] [YSQL] Fix metric for rejections due to connections limit
• [16401] [yugabyted] configure data_placement not working with constraint value with uppercases
• [16444] [DOCDB] Fix tsan and fastdebug test, Ignore heartbeats from peers not in reported config
• [16451] [YSQL] Fix imports in D22790 and D22956
• [16481] [CDCSDK] [Colocation] YBClient fails to map multiple colocated tables to same tablet
• [16518] [DOCDB] Prevent resuming conflict resolution at special timestamps in case transaction is not found
• [16526] [DOCDB] Enable remote bootstrap to fix missing tablet replica problem in CreateTableStressTest::TestConcurrentCreateTableAndReloadMetadata.
• [16540] [YSQL] Address HLL ASAN failures
• [16541] [yugabyted] Show EAR in status if enabled even when --secure flag is false
• [16552] [YSQL] Prevent prefix_length_ from taking non-key columns into account
• [16565] [DOCDB] Fixed missing data in case of RocksDB iterator error
• [16570] [YSQL] Fix build errors when building with --no-tcmalloc
• [16571] [DOCDB] Fix InboundCall::trace() regression
• [16583] [YSQL] Fix IndexOnlyScan with no targets returning empty rows
• [16615] Dev tools: Fix script compatibility with newer arcanist
• [16626] [DOCDB] Fix TSAN failure:XClusterConsumer and XClusterOutputClient deadlock
• [16632] [DOCDB] Fix TestIndex.testDropDuringWrite
• [16637] [CDCSDK] Fix safe_time issues around large transactions
• [16644] [DST] PITR Fix YbAdminSnapshotScheduleTest.PgsqlAddColumnCompactWithPackedRow test
• [16669] [DOCDB] Fix tablet splitting / PITR deadlock
• [16679] [CDCSDK] Correcting safe_time bug for batches with only commit record
• [16713] [YSQL] Fix heap-buffer-overflow in ysql_upgrade.cc
• [16745] [YSQL] Fix TestPgRegressAuthorization failure
• [16748] [DOCDB] Fix colocation parent table's metrics update when a colocated table is altered
• [16763] [xCluster] Reduce logging noise during xcluster errors
• [16773] [xCluster] Fix TSAN race on StreamTabletMetadata
• [16790] [CDCSDK] GetCheckpoint will return invalid checkpoint for unpolled tablets
• [16807] [DOCDB] Enable automatic_tablet_splitting for new clusters using AutoFlags
• [16851] Fix DB UT for learner-TS due to broadcast address changes in yb_backup.py
• [16944] [16715] [DOCDB] Fix 2 bugs with xcluster + packed, Fixing the demo connect creating tables to default database.
• [16964] [DOCDB] Compilation failure for Apple clang 14.0.3
• [16976] [YSQL] Fix postgres crash when pg_stat_progress_create_index has an index from a different database
• [16992] [YSQL] Fix scan upperbound calculation during batched IN query on hash columns
• [16995] [CDCSDK] Removed duplicated access to CDC Stream table
• [17067] [YSQL] Disallow hash batching if there is only one batch element
• [17118] [CDCSDK] Remove tables with no active streams from namespace_to_cdcsdk_unprocessed_table_map_
• [17150] [YSQL] Disable batching on join clauses that involve more than one base relation on a side
• [17165] [CDCSDK] Added serve_as_proxy member to GetCheckpointRequestPB
• [17253] [DocDB] Fix RPC error handling in pg_client
• [CLOUDGA-12305] Missing getters for provider fields - Part 2
• [CLOUDGA-13117] Remove the OS update playbook components
• [YSQL] style: fix/improve most of D21043 style
• Fix xCluster utest build error

Known issues

N/A