Disabling tests

This document describes how to disable a test from running in the continuous integration (CI) test system.

Why disable tests?

Tests are either disabled permanently or temporarily:

• Permanently: the test is never expected to run in a certain configuration, due to the design of the test or product.
• Temporarily: the test is failing, and needs to be disabled so a test run isn't continuously made "noisy" by the existence of a failing test in the results. These tests are expected to be re-enabled when a bug is fixed or feature is implemented.

Runtime or libraries?

There are two main sets of tests in the repo: runtime tests in the src/tests tree, and libraries tests which are spread amongst the libraries in the src/libraries tree. (There are also PAL tests in src/coreclr/pal/tests, which are ignored here.)

The two types have similar mechanisms for disabling, but the runtime tests have more options given how they build and execute.

Test configuration

You need to determine under which configuration you wish to disable the test:

• For all configurations
• For just one processor architecture (x86, x64, arm32, arm64)
• For just one runtime (coreclr, mono) or mono runtime variant (monointerpreter, llvmaot, llvmfullaot)
• For just one operating system (Windows, Linux, macOS, Android, iOS)
• For a particular run type:
  • GCStress
  • JIT stress (any type)
  • ildasm/ilasm round-trip testing
  • ReadyToRun testing

Generally, you should disable a test under the most specific condition that is causing the test failure. Thus, if the test only fails on arm64, don't disable it for all architectures. If the test only fails on macOS, don't disable it for Windows or Linux.

If it is unclear the full set configurations where the test is failing, it is sometimes necessary and expedient to disable the test more broadly than possibly required.

Disabling runtime tests (src/tests)

Disabling runtime tests (src/tests) with xunit attributes

The runtime tests use an XUnit-based model for test execution. There are a number of attributes provided for filtering based on different test modes. Here are some examples of attributes that can be applied to tests to prevent them from running in certain configurations:

• Prevent a test from running on Mono: [SkipOnMono]
• Prevent a test from running on CoreCLR: [SkipOnCoreClr]
• Prevent a test from running on NativeAOT: [ConditionalFact("", typeof(TestLibrary.Utilities), nameof(Utilities.IsNotNativeAot))] instead of [Fact]
• Prevent a test from running under GCStress: [SkipOnCoreClr("Reason", RuntimeTestModes.AnyGCStress)]
• Prevent a test from running under HeapVerify: [SkipOnCoreClr("Reason", RuntimeTestModes.HeapVerify)]
• Prevent a test from running under JIT stress modes: [SkipOnCoreClr("Reason", RuntimeTestModes.AnyJitStress)]

Additionally, the ConditionalFact, ConditionalTheory, PlatformSpecific, and ActiveIssue attributes are available for usage to disable or enable tests only on specific platforms or configurations.

Some test modes are processed at the assembly level. For these tests, you should mark the tests as <RequiresProcessIsolation>true</RequiresProcessIsolation> and set one of the attributes in the following section.

Disabling runtime tests (src/tests) with CLRTestTargetUnsupported

Generally, tests should be diabled via xunit attributes unless it is not possible to do so. There are a few scenarios where this might occur, all in cases where the test project is marked with <RequiresProcessIsolation>true</RequiresProcessIsolation>:

• A test may have an explicit Main method and not use the XUnitWrapperGenerator. In this case, the XUnit attributes will not apply.
• A test may be written in a non-C# language (generally IL) and be required to have <RequiresProcessIsolation>true</RequiresProcessIsolation> for another reason. In this case, there is no generator.

In this case, the test should be disabled by setting the CLRTestTargetUnsupported property in its project file with a relevant condition and a comment with the issue link or reason.

Disabling runtime tests (src/tests) with test configuration properties

Some test configurations must be disabled by editing the .csproj or .ilproj file for the test, and inserting a property in a <PropertyGroup>, as follows:

• Prevent a test from running when testing unloadability: add <UnloadabilityIncompatible>true</UnloadabilityIncompatible>
• Prevent a test from running when testing ildasm/ilasm round-tripping: add <IlasmRoundTripIncompatible>true</IlasmRoundTripIncompatible>
• Prevent a test assembly from being passed to the Mono AOT compiler: add <MonoAotIncompatible>true</MonoAotIncompatible>
• Prevent a test from being passed to CrossGen2: add <CrossGenTest>false</CrossGenTest>
• Prevent a test from being passed to the NativeAOT ILCompiler and run under NativeAOT: add <NativeAotIncompatible>true</NativeAotIncompatible>

When one of the following settings is already required for a given test, the following settings can also be set in the project file instead of via XUnit attributes. They should not be the only reason a test is marked as <RequiresProcessIsolation>true</RequiresProcessIsolation>, however.

• Prevent a test from running under GCStress: add <GCStressIncompatible>true</GCStressIncompatible>
• Prevent a test from running running under JIT stress modes: add <JitOptimizationSensitive>true</JitOptimizationSensitive>
• Prevent a test from running under HeapVerify: add <HeapVerifyIncompatible>true</HeapVerifyIncompatible>

Note that these properties can be conditional, e.g.:

<GCStressIncompatible Condition="'$(TargetArchitecture)' == 'arm64' and '$(TargetOS)' == 'osx'">true</GCStressIncompatible>

(REVIEW: I'm not clear which conditions are allowed, and respected.)

More information about writing/adding tests to src/tests can be found here.

Disabling libraries tests (src/libraries)

Information on disabling libraries tests is found here.

In particular, look at ActiveIssueAttribute, SkipOnCoreClrAttribute, and SkipOnMonoAttribute.