ContextQMD
Back to Mongo

Multiversion Testing

docs/evergreen-testing/multiversion.md

3.6.17-windows-splunk-v313.7 KB
Original Source

Multiversion Testing

Table of contents

Terminology and overview

Introduction

Some tests test specific upgrade/downgrade behavior expected between different versions of MongoDB. Several versions of MongoDB are spun up during those test runs.

  • Multiversion suites - resmoke suites that are running tests with several versions of MongoDB.

  • Multiversion tasks - Evergreen tasks that are running multiversion suites. Multiversion tasks in most cases include multiversion or downgrade in their names.

Latest vs last-lts vs last-continuous

For some of the versions we are using such generic names as latest, last-lts and last-continuous.

  • latest - the current version. In Evergreen, the version that was compiled in the current build.

  • last-lts - the latest LTS (Long Term Support) Major release version. In Evergreen, the version that was downloaded from the last LTS release branch project. It resolves to an entry in longTermSupportReleases of releases.yml.

  • last-continuous - the latest Rapid release version. In Evergreen, the version that was downloaded from the Rapid release branch project. It resolves to the entry in featureCompatibilityVersions of releases.yml that looks older than the output of git describe. Will not be tested against if it is listed in eolVersions as being end of life.

Note: The latest release.yml file from master is always used, even fetched remotely when on another branch.

Old vs new

Many multiversion tasks are running tests against latest/last-lts or latest/last-continuous versions. In such context we refer to last-lts and last-continuous versions as the old version and to latest as a new version.

A new version is compiled in the same way as for non-multiversion tasks. The old versions of compiled binaries are downloaded from the old branch projects with db-contrib-tool. db-contrib-tool searches for the latest available compiled binaries on the old branch projects in Evergreen.

Explicit and Implicit multiversion suites

Multiversion suites can be explicit and implicit.

js
const versions = [
  {
    binVersion: "4.4",
    featureCompatibilityVersion: "4.4",
    testCollection: "four_four",
  },
  {
    binVersion: "5.0",
    featureCompatibilityVersion: "5.0",
    testCollection: "five_zero",
  },
  {
    binVersion: "6.0",
    featureCompatibilityVersion: "6.0",
    testCollection: "six_zero",
  },
  {
    binVersion: "last-lts",
    featureCompatibilityVersion: lastLTSFCV,
    testCollection: "last_lts",
  },
  {
    binVersion: "last-continuous",
    featureCompatibilityVersion: lastContinuousFCV,
    testCollection: "last_continuous",
  },
  {
    binVersion: "latest",
    featureCompatibilityVersion: latestFCV,
    testCollection: "latest",
  },
];
  • Implicit - JS tests know nothing about the binary versions they are running, e.g. retryable_writes_downgrade.yml. Most of the implicit multiversion suites are using matrix suites, e.g. replica_sets_last_lts:
bash
$ python buildscripts/resmoke.py suiteconfig --suite=replica_sets_last_lts

executor:
  config:
    shell_options:
      global_vars:
        TestData:
          useRandomBinVersionsWithinReplicaSet: last-lts
      nodb: ''
selector:
  exclude_files:
  - jstests/replsets/initial_sync_rename_collection.js
  - jstests/replsets/initial_sync_drop_collection.js
  - jstests/replsets/apply_prepare_txn_write_conflict_robustness.js
  - jstests/replsets/invalidate_sessions_on_stepdown.js
  - jstests/replsets/initial_sync_fails_unclean_restart.js
  exclude_with_any_tags:
  - multiversion_incompatible
  - backport_required_multiversion
  - replica_sets_multiversion_backport_required_multiversion
  - disabled_for_fcv_6_1_upgrade
  roots:
  - jstests/replsets/*.js
test_kind: js_test

In implicit multiversion suites the version of binaries is defined on the resmoke fixture level.

The example of replica set fixture configuration override:

yaml
fixture:
  num_nodes: 3
  old_bin_version: last_lts
  mixed_bin_versions: new_new_old

The example of sharded cluster fixture configuration override:

yaml
fixture:
  num_shards: 2
  num_rs_nodes_per_shard: 2
  old_bin_version: last_lts
  mixed_bin_versions: new_old_old_new

The example of shell fixture configuration override:

yaml
value:
  executor:
    config:
      shell_options:
        global_vars:
          TestData:
            useRandomBinVersionsWithinReplicaSet: "last-lts"

Version combinations

In implicit multiversion suites the same set of tests may run in similar suites that are using various mixed version combinations. Those version combinations depend on the type of resmoke fixture the suite is running with. These are the recommended version combinations to test against based on the suite fixtures:

If last-lts and last-continuous versions happen to be the same, or last-continuous is EOL, we skip last-continuous and run multiversion suites with only last-lts combinations in Evergreen.

Working with multiversion tasks in Evergreen

Multiversion task generation

Please refer to mongo-task-generator documentation for generating multiversion tasks in Evergreen.

Exclude tests from multiversion testing

Sometimes tests are not designed to run in multiversion suites. To avoid implicit multiversion suites to pick up the test multiversion_incompatible tag can be added to the test, e.g. jstests/concurrency/fsm_workloads/drop_database_sharded_setFCV.js.

js
// @tags: [multiversion_incompatible]

There is additional requires_fcv_XX set of tags that can be used to disable tests from running in multiversion where XX is the version number, e.g. requires_fcv_70 stands for version 7.0 in jstests/auth/user_roles_empty.js.

js
// @tags: [requires_fcv_61]

Tests with requires_fcv_XX tags are excluded from multiversion tasks that may run the versions below the specified FCV version, e.g. when the latest version is 6.2, last-continuous is 6.1 and last-lts is 6.0, tests tagged with requires_fcv_61 will NOT run in multiversion tasks that run latest with last-lts, but will run in multiversion tasks that run lastest with last-continuous.

In addition to disabling multiversion tests based on FCV, there is no need to run in-development featureFlagXYZ tests (featureFlags that have default: false) because these tests will most likely fail on older versions that have not implemented this feature. For multiversion tasks, we pass the --runNoFeatureFlagTests flag to avoid these failures on all feature flag variants.

For more info on FCV, take a look at FCV_AND_FEATURE_FLAG_README.md.

Another common case could be that the changes on master branch are breaking multiversion tests, but with those changes backported to the older branches the multiversion tests should work. In order to temporarily disable the test from running in multiversion it can be added to the etc/backports_required_for_multiversion_tests.yml. Please follow the instructions described in the file.