.agents/skills/testing-internals/SKILL.md
This document explains tests.cmd internals and helps troubleshoot test execution issues.
The test execution chain:
tests.cmd → Bazel → IdeaUltimateRunTestsBuildTarget → TestingTasksImpl → JUnit 5
Key components:
//build:idea_ultimate_run_tests_build_target-Dintellij.build.test.* system propertiesSymptoms:
Causes & Solutions:
Pattern mismatch - Simple class names don't work (see Pattern Matching below):
# WRONG - simple name won't match FQN
-Dintellij.build.test.patterns=MyTest
# CORRECT - use wildcard or FQN
-Dintellij.build.test.patterns=*MyTest
-Dintellij.build.test.patterns=com.example.MyTest
Test not in classpath - The test class must be in a module that's part of the test classpath. Check if the module is included in the build.
Test class not recognized - Ensure class name ends with Test or has JUnit annotations.
Solution: Increase heap size:
./tests.cmd \
-Dintellij.build.test.patterns=MyTest \
-Dintellij.build.test.jvm.memory.options=-Xmx8g
Symptoms:
Solutions:
Check module dependencies - Ensure test module has required dependencies in .iml file.
Verify BUILD.bazel is synced - Run ./build/jpsModelToBazel.cmd after changing .iml files.
Note: Bazel incremental builds are always correct. Do not use bazel clean - it won't help.
Symptoms:
Debug steps:
Check test groups - If using groups, verify testGroups.properties configuration.
Check bucketing - For parallel execution, tests are distributed by hash:
# See which bucket a test falls into
-Didea.test.runners.count=4
-Didea.test.runner.index=0 # Run only bucket 0
Check class filters - TestCaseLoader applies pattern matching before test execution.
Enable debug mode to attach a debugger:
./tests.cmd \
-Dintellij.build.test.patterns=MyTest \
-Dintellij.build.test.debug.enabled=true \
-Dintellij.build.test.debug.port=5005 \
-Dintellij.build.test.debug.suspend=true
Then attach debugger to port 5005.
| Property | Purpose |
|---|---|
intellij.build.test.patterns | Test class patterns (semicolon-separated) |
intellij.build.test.groups | Test groups to run |
intellij.build.test.attempt.count | Retry count for flaky tests |
intellij.build.test.jvm.memory.options | JVM memory settings |
intellij.build.test.debug.enabled | Enable remote debugging |
┌─────────────────────────────────────────────────────────────────────────────┐
│ 1. COMMAND LINE │
│ ./tests.cmd -Dintellij.build.test.patterns=MyTest │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 2. SHELL SCRIPT │
│ tests.cmd → community/build/run_build_target.sh │
│ Converts args to --jvm_flag=<arg> format │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 3. BAZEL │
│ bazel run //build:idea_ultimate_run_tests_build_target │
│ (defined in build/BUILD.bazel) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 4. BUILD TARGET ENTRY POINT │
│ IdeaUltimateRunTestsBuildTarget.main() │
│ → UltimateProjectTestingTasks.runTests() │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 5. TEST OPTIONS PARSING │
│ UltimateProjectTestingOptions (extends TestingOptions) │
│ Reads all -Dintellij.build.test.* system properties │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 6. TEST EXECUTION ORCHESTRATION │
│ TestingTasksImpl.runTests() │
│ - Builds test classpath │
│ - Prepares JVM arguments and system properties │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 7. FORKED TEST PROCESS │
│ TestingTasksImpl.runJUnit5Engine() │
│ Spawns new JVM with bootstrap classpath │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌───────────────────────────────────┐ ┌───────────────────────────────────────┐
│ 8a. JUNIT 5 TESTS │ │ 8b. JUNIT 3/4 TESTS (Legacy) │
│ JUnit5TeamCityRunner.main() │ │ JUnit5TeamCityRunner.main() │
│ - Uses JUnit Platform Launcher │ │ - Uses JUnit Platform Launcher │
│ - ClassNameFilter │ │ - ClassNameFilter │
│ - PostDiscoveryFilter │ │ - PostDiscoveryFilter │
└───────────────────────────────────┘ └───────────────────────────────────────┘
│ │
└───────────────┬───────────────┘
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 9. TEST DISCOVERY & FILTERING │
│ TestCaseLoader │
│ - Loads test classes from classpath roots │
│ - Applies pattern/group filters │
│ - Handles bucketing for parallel execution │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 10. TEST EXECUTION │
│ JUnit Platform executes tests │
│ TCExecutionListener reports results to TeamCity │
└─────────────────────────────────────────────────────────────────────────────┘
| Class | Purpose |
|---|---|
IdeaUltimateRunTestsBuildTarget | Ultimate tests entry point, calls UltimateProjectTestingTasks |
CommunityRunTestsBuildTarget | Community tests entry point, calls TestingTasks |
UltimateProjectTestingTasks | Ultimate-specific test orchestration (YourKit, network restrictions) |
TestingTasks | Interface for test execution |
TestingTasksImpl | Core test execution logic |
| Class | Purpose |
|---|---|
TestingOptions | Base class for all test options. Parses -Dintellij.build.test.* properties |
UltimateProjectTestingOptions | Ultimate-specific options (YourKit, skip community tests) |
| Class | Purpose |
|---|---|
JUnit5TeamCityRunner | Runs JUnit 3/4 tests using the JUnit Vintage test engine, or JUnit5 tests using the JUnit Jupiter test engine |
TCExecutionListener | Reports test results to TeamCity via service messages |
| Class | Purpose |
|---|---|
TestCaseLoader | Discovers and filters test classes |
TestAll | JUnit 3 test suite, collects all tests |
TestClassesFilter | Pattern/group-based test filtering |
| Class | Purpose |
|---|---|
BucketingScheme | Interface for test distribution |
HashingBucketingScheme | Default: hash-based distribution |
TestsDurationBucketingScheme | Duration-aware distribution |
| Product | Entry Point | Default mainModule | Source |
|---|---|---|---|
| IDEA Ultimate | IdeaUltimateRunTestsBuildTarget | intellij.idea.ultimate.tests.main | build/src/ |
| Community | CommunityRunTestsBuildTarget | intellij.idea.community.main.tests | community/build/src/ |
| RustRover | RustRoverRunTestsBuildTarget | intellij.idea.ultimate.tests.main | rustrover/build/src/ |
| RubyMine | RubyRunTestsBuildTarget | intellij.idea.ultimate.tests.main | ruby/build/src/ |
| CLion | CLionRunTestsBuildTarget | intellij.idea.ultimate.tests.main | CIDR/clion-build/src/ |
Note: Product entry points (RustRover, RubyMine, CLion) inherit intellij.idea.ultimate.tests.main as the default, but to run product-specific tests, use the dedicated test module with -Dintellij.build.test.main.module. See TESTING.md for the correct module per product.
From intellij-teamcity-config/.teamcity/src/ijplatform/KnownModules.kt:
| CI Constant | Module Name |
|---|---|
ULTIMATE_TESTS | intellij.idea.ultimate.tests.main |
COMMUNITY_MAIN | intellij.idea.community.main.tests |
GOLAND_TESTS | intellij.goland.tests |
PYTHON_TESTS | intellij.python.tests |
PHPSTORM_MAIN | intellij.phpstorm.main.tests |
CLION_MAIN | intellij.clion.main.tests |
RUSTROVER_MAIN | intellij.rustrover.main.tests |
KOTLIN_K2_TESTS | kotlin.fir-all-tests |
KOTLIN_ULTIMATE_ALL_TESTS | intellij.kotlin-ultimate.all-tests |
DATABASE_TESTS | intellij.database.tests |
DATABASE_SQL_TESTS | intellij.database.sql.tests |
Default mainModule is set in:
UltimateProjectTestingOptions.kt:36 - Ultimate: intellij.idea.ultimate.tests.mainCommunityRunTestsBuildTarget.kt:28 - Community: intellij.idea.community.main.testsintellij.idea.ultimate.tests.main
├── intellij.idea.ultimate.tests
├── intellij.idea.ultimate.tests.kotlin
├── intellij.platform.tests
├── intellij.java.tests
└── ... (hundreds of test modules)
Separate hierarchies (NOT in .main):
├── intellij.idea.ultimate.tests.kotlin.k2
│ └── intellij.devkit.kotlin.fir.tests
├── intellij.idea.ultimate.tests.devBuildTests
└── kotlin.fir-all-tests (K2/FIR tests)
Most options use the intellij.build.test.* prefix. Bucketing uses idea.test.* prefix.
// Test selection (mutually exclusive, in priority order)
testConfigurations // -Dintellij.build.test.configurations=<config>
testPatterns // -Dintellij.build.test.patterns=<pattern>
testGroups // -Dintellij.build.test.groups=<group>
// Test execution
mainModule // -Dintellij.build.test.main.module=<module>
attemptCount // -Dintellij.build.test.attempt.count=<n>
// JVM configuration
jvmMemoryOptions // -Dintellij.build.test.jvm.memory.options=<opts>
customRuntimePath // -Dintellij.build.test.jre=<path>
// Debugging
isDebugEnabled // -Dintellij.build.test.debug.enabled=<bool>
debugPort // -Dintellij.build.test.debug.port=<port>
isSuspendDebugProcess // -Dintellij.build.test.debug.suspend=<bool>
// Bucketing (parallel execution) - NOTE: uses idea.test.* prefix
bucketsCount // -Didea.test.runners.count=<n>
bucketIndex // -Didea.test.runner.index=<n>
// Coverage
enableCoverage // -Dintellij.build.test.coverage.enabled=<bool>
coveredClassesPatterns // -Dintellij.build.test.coverage.include.class.patterns=<patterns>
The test target is defined in build/BUILD.bazel:
java_binary(
name = "idea_ultimate_run_tests_build_target",
runtime_deps = [":build"],
main_class = "IdeaUltimateRunTestsBuildTarget",
data = ALL_ULTIMATE_TARGETS,
jvm_flags = [
"-Dintellij.build.console.exporter.to.temp.file=true",
"-Dintellij.build.console.messages.verbose=false",
"-Dintellij.build.clean.output.root=false", # Reuse compiled classes
"-Dintellij.build.use.compiled.classes=true", # Skip recompilation
],
add_opens = INTELLIJ_ADD_OPENS,
)
TestingTasksImpl.prepareEnvForTestRun() configures the forked test JVM:
// Key system properties set for test process:
"idea.home.path" → projectHome
"idea.config.path" → tempDir/config
"idea.system.path" → tempDir/system
"java.io.tmpdir" → tempDir
// JVM options:
"-XX:+HeapDumpOnOutOfMemoryError"
"-XX:HeapDumpPath=<snapshotsDir>/intellij-tests-oom-<timestamp>.hprof"
"-Xms750m -Xmx1024m" // or custom from jvmMemoryOptions
// Plus --add-opens for module access
There are two mechanisms for passing JVM arguments to the test JVM process:
intellij.build.test.jvm.memory.options)For JVM memory settings like heap size, use the dedicated property:
./tests.cmd -Dintellij.build.test.jvm.memory.options="-Xmx4g -Xms2g"
Multiple options are space-separated within quotes. These options are added to the beginning of the JVM arguments via VmOptionsGenerator.generate().
Implementation (see TestingTasksImpl.kt, runJUnit5Engine method):
val customMemoryOptions = options.jvmMemoryOptions?.trim()?.split(Regex("\\s+"))?.takeIf { it.isNotEmpty() }
jvmArgs.addAll(
index = 0,
elements = VmOptionsGenerator.generate(
customVmMemoryOptions = if (customMemoryOptions == null) mapOf("-Xms" to "750m", "-Xmx" to "1024m") else emptyMap(),
additionalVmOptions = customMemoryOptions ?: emptyList(),
// ... other parameters omitted
),
)
pass.* prefix)To pass arbitrary system properties to the test JVM, use the pass. prefix. The prefix is stripped before passing to the test process:
./tests.cmd -Dpass.my.custom.property=value -Dpass.some.flag=true
Results in test JVM receiving:
-Dmy.custom.property=value-Dsome.flag=trueImplementation (see TestingTasksImpl.kt, prepareEnvForTestRun method):
for ((key, value) in System.getProperties()) {
key as String
if (key.startsWith("pass.")) {
systemProperties.put(key.substring("pass.".length), value as String)
}
}
This is a TeamCity convention for passing properties to nested processes.
./tests.cmd \
-Dintellij.build.test.patterns=MyTest \
-Dintellij.build.test.jvm.memory.options="-Xmx4g" \
-Dpass.my.test.flag=enabled \
-Dpass.debug.level=verbose
Important: Properties without pass. prefix are consumed by the build scripts, NOT passed to the test JVM.
┌─────────────────────────────────────────────────────────────────────────────┐
│ PATTERN INPUT │
│ -Dintellij.build.test.patterns=*MyTest │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ PATTERN COMPILATION (TestClassesFilter.compilePattern) │
│ filter.replace("$","\\$").replace(".","\\.");replace("*",".*") │
│ "*MyTest" → ".*MyTest" │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ CLASS DISCOVERY (ClassFinder) │
│ Scans JARs for *Test.class files, extracts FQN: │
│ org/example/MyTest.class → "org.example.MyTest" │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ PATTERN MATCHING (PatternListTestClassFilter.matches) │
│ pattern.matcher(className).matches() ← FULL STRING MATCH │
│ ".*MyTest".matches("org.example.MyTest") → true │
└─────────────────────────────────────────────────────────────────────────────┘
| Input Pattern | Compiled Regex | Matches org.example.MyTest? |
|---|---|---|
MyTest | MyTest | ❌ NO (not full match) |
*MyTest | .*MyTest | ✅ YES |
org.example.MyTest | org\.example\.MyTest | ✅ YES |
org.example.* | org\.example\..* | ✅ YES |
* | .* | ✅ YES (matches all) |
Pattern Compilation (TestClassesFilter.compilePattern()):
filter = filter.replace("$", "\\$").replace(".", "\\.").replace("*", ".*");
return Pattern.compile(filter);
Pattern Matching (PatternListTestClassFilter.matches()):
return ContainerUtil.exists(patterns, pattern -> pattern.matcher(className).matches());
matches() NOT find() - requires ENTIRE string to matchClassNameFilter (JUnit 5) - Fast pre-filter on class names
TestCaseLoader.isClassNameIncluded(className)PostDiscoveryFilter (JUnit 5) - Post-discovery filter
TestCaseLoader.isClassIncluded(className)TestCaseLoader.fillTestCases() (JUnit 3/4) - Scans classpath roots
ClassFinder to find all *Test.class filesisPotentiallyTestCase() which calls filter's matches()testGroups.propertiesRoot Cause: Pattern.matches() requires the ENTIRE string to match.
// Pattern: "MyTest" → Regex: "MyTest"
// className: "org.example.MyTest"
"MyTest".matches("org.example.MyTest") // Returns FALSE
// Pattern: "*MyTest" → Regex: ".*MyTest"
".*MyTest".matches("org.example.MyTest") // Returns TRUE
This applies to ALL modules (default and non-default). Always use:
org.example.MyTest*MyTesttests.cmd (quick start, parameters, examples)@TestApplication, fixtures, EDT)