ContextQMD
Back to Opensearch

Opensearch.Release Notes 2.10.0

release-notes/opensearch.release-notes-2.10.0.md

3.6.017.4 KB
Original Source

2023-09-08 Version 2.10.0 Release Notes

[2.10]

Added

  • Add server version as REST response header #6583
  • Start replication checkpointTimers on primary before segments upload to remote store. (#8221https://github.com/opensearch-project/OpenSearch/pull/8221)
  • Introduce new static cluster setting to control slice computation for concurrent segment search. (#8847)
  • Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache (#8606)
  • Disallow compression level to be set for default and best_compression index codecs (#8737https://github.com/opensearch-project/OpenSearch/pull/8737)
  • [distribution/archives] [Linux] [x64] Provide the variant of the distributions bundled with JRE (#8195https://github.com/opensearch-project/OpenSearch/pull/8195)
  • Prioritize replica shard movement during shard relocation (#8875)
  • Introducing Default and Best Compression codecs as their algorithm name (#9123)
  • Make SearchTemplateRequest implement IndicesRequest.Replaceable (#9122)
  • [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (#9223)
  • [Remote Store] Add Segment download stats to remotestore stats API (#8718)
  • [Remote Store] Add remote segment transfer stats on NodesStats API (#9168 #9393 #9454)
  • [Segment Replication] Support realtime reads for GET requests (#9212)
  • Allow test clusters to run with TLS (#8900)
  • Add jdk.incubator.vector module support for JDK 20+ (#8601)
  • [Feature] Expose term frequency in Painless script score context (#9081)
  • Add support for reading partial files to HDFS repository (#9513)
  • [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
  • [BWC and API enforcement] Decorate the existing APIs with proper annotations (part 1) (#9520)
  • Add support for extensions to search responses using SearchExtBuilder (#9379)
  • [Remote State] Create service to publish cluster state to remote store (#9160)
  • Core crypto library to perform encryption and decryption of source content (#8466)
  • Expose DelimitedTermFrequencyTokenFilter to allow providing term frequencies along with terms (#9479)
  • APIs for performing async blob reads and async downloads from the repository using multiple streams (#9592)
  • Add concurrent segment search related metrics to node and index stats (#9622)
  • Add average concurrency metric for concurrent segment search (#9670)
  • Introduce cluster default remote translog buffer interval setting (#9584)
  • Added encryption-sdk lib to provide encryption and decryption capabilities (#8466 #9289)
  • [Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level (#9709)
  • Added crypto-kms plugin to provide AWS KMS based key providers for encryption/decryption. (#8465)
  • [Remote state] Integrate remote cluster state in publish/commit flow (#9665)
  • [Remote Store] Changes to introduce repository registration during bootstrap via node attributes. (#9105)
  • [Remote state] Auto restore index metadata from last known cluster state (#9831)

Dependencies

  • Bump org.apache.logging.log4j:log4j-core from 2.17.1 to 2.20.0 (#8307)
  • Bump io.grpc:grpc-context from 1.46.0 to 1.57.1 (#8726, #9145)
  • Bump com.netflix.nebula:gradle-info-plugin from 12.1.5 to 12.1.6 (#8724)
  • Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#8725)
  • Bump org.apache.zookeeper:zookeeper from 3.8.1 to 3.9.0 (#8844, #9146)
  • Bump org.gradle.test-retry from 1.5.3 to 1.5.4 (#8842)
  • Bump com.netflix.nebula.ospackage-base from 11.3.0 to 11.4.0 (#8838)
  • Bump com.google.http-client:google-http-client-gson from 1.43.2 to 1.43.3 (#8840)
  • OpenJDK Update (July 2023 Patch releases) (#8869)
  • Bump hadoop libraries from 3.3.4 to 3.3.6 (#6995)
  • Bump com.gradle.enterprise from 3.13.3 to 3.14.1 (#8996)
  • Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 (#8995)
  • Bump com.google.cloud:google-cloud-core-http from 2.21.0 to 2.21.1 (#8999)
  • Bump com.maxmind.geoip2:geoip2 from 4.0.1 to 4.1.0 (#8998)
  • Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 in /plugins/repository-hdfs (#8997)
  • Bump netty from 4.1.94.Final to 4.1.96.Final (#9030)
  • Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#9080)
  • Bump io.projectreactor.netty:reactor-netty-http from 1.1.8 to 1.1.9 (#9147)
  • Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 (#9148)
  • Bump com.azure:azure-storage-blob from 12.22.3 to 12.23.0 (#9231)
  • Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (#9227)
  • Bump org.xerial.snappy:snappy-java from 1.1.8.2 to 1.1.10.3 (#9252)
  • Bump com.squareup.okhttp3:okhttp from 4.9.3 to 4.11.0 (#9252)
  • Bump com.squareup.okio:okio from 2.8.0 to 3.5.0 (#9252)
  • Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 (#9230)
  • Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (#9228)
  • Bump snakeyaml from 2.0 to 2.1 (#9269)
  • Bump aws-actions/configure-aws-credentials from 1 to 2 (#9302)
  • Bump com.github.luben:zstd-jni from 1.5.5-3 to 1.5.5-5 (#9431
  • Bump netty from 4.1.96.Final to 4.1.97.Final (#9553)
  • Bump io.grpc:grpc-api from 1.57.1 to 1.57.2 (#9578)
  • Add Encryption SDK dependencies (#8466)

Changed

  • Default to mmapfs within hybridfs (#8508)
  • Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
  • Make Span exporter configurable (#8620)
  • Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
  • [Refactor] StreamIO from common to core.common namespace in core lib (#8157)
  • [Refactor] Remaining HPPC to java.util collections (#8730)
  • Remote Segment Store Repository setting moved from index.remote_store.repository to index.remote_store.segment.repository and cluster.remote_store.repository to cluster.remote_store.segment.repository respectively for Index and Cluster level settings (#8719)
  • Change InternalSignificantTerms to sum shard-level superset counts only in final reduce (#8735)
  • Exclude 'benchmarks' from codecov report (#8805)
  • Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory (#8807)
  • Replace the deprecated IndexReader APIs with new storedFields() & termVectors() (#7792)
  • [Remote Store] Add support to restore only unassigned shards of an index (#8792)
  • Add safeguard limits for file cache during node level allocation (#8208)
  • Performance improvements for BytesRefHash (#8788)
  • Add support for aggregation profiler with concurrent aggregation (#8801)
  • [Remove] Deprecated Fractional ByteSizeValue support #9005 (#9005)
  • Add support for aggregation profiler with concurrent aggregation (#8801)
  • [Remote Store] Restrict user override for remote store index level settings (#8812)
  • [Refactor] MediaTypeParser to MediaTypeParserRegistry (#8636)
  • Make MultiBucketConsumerService thread safe to use across slices during search (#9047)
  • Removed blocking wait in TransportGetSnapshotsAction which was exhausting generic threadpool (#8377)
  • Adds support for tracing runnable scenarios (#8831)
  • Change shard_size and shard_min_doc_count evaluation to happen in shard level reduce phase (#9085)
  • Add attributes to startSpan methods (#9199)
  • [Refactor] Task foundation classes to core library - pt 1 (#9082)
  • Add support for wrapping CollectorManager with profiling during concurrent execution (#9129)
  • Add base class for parameterizing the search based tests #9083 (#9083)
  • Add support for wrapping CollectorManager with profiling during concurrent execution (#9129)
  • Rethrow OpenSearch exception for non-concurrent path while using concurrent search (#9177)
  • Improve performance of encoding composite keys in multi-term aggregations (#9412)
  • Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility (#9262)
  • Fix sort related ITs for concurrent search (#9177
  • [Remote Store] Implicitly use replication type SEGMENT for remote store clusters (#9264)
  • Add support to use trace propagated from client (#9506)
  • Separate request-based and settings-based concurrent segment search controls and introduce AggregatorFactory method to determine concurrent search support (#9469)
  • [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
  • [Remote Store] Implicitly use replication type SEGMENT for remote store clusters (#9264)
  • Redefine telemetry context restoration and propagation (#9617)
  • Use non-concurrent path for sort request on timeseries index and field(#9562)
  • Added sampler based on Blanket Probabilistic Sampling rate and Override for on demand (#9621)
  • Decouple replication lag from logic to fail stale replicas (#9507)
  • Improve performance of rounding dates in date_histogram aggregation (#9727)
  • [Remote Store] Add support for Remote Translog Store stats in _remotestore/stats/ API (#9263)
  • Removing the vec file extension from INDEX_STORE_HYBRID_NIO_EXTENSIONS, to ensure the no performance degradation for vector search via Lucene Engine.(#9528))
  • Cleanup Unreferenced file on segment merge failure (#9503)
  • Move zstd compression codec to external custom-codecs repository (#9422
  • [Remote Store] Add support for Remote Translog Store upload stats in _nodes/stats/ API (#8908)
  • [Remote Store] Removing feature flag to mark feature GA (#9761)

Removed

  • Remove provision to create Remote Indices without Remote Translog Store (#8719)

Fixed

  • Fix flaky ResourceAwareTasksTests.testBasicTaskResourceTracking test (#8993)
  • Fix null_pointer_exception when creating or updating ingest pipeline (#9259)
  • Fix memory leak when using Zstd Dictionary (#9403)
  • Fix condition to remove index create block (#9437)
  • Add support to clear archived index setting (#9019)
  • Fix range reads in respository-s3 (9512)
  • [Segment Replication] Fixed bug where replica shard temporarily serves stale data during an engine reset (#9495)
  • Disable shard/segment level search_after short cutting if track_total_hits != false (#9683)
  • [Segment Replication] Fixed bug where bytes behind metric is not accurate (#9686)