ContextQMD
Back to Swift Package Manager

Swift Package Manager Tests with `withKnownIssue` Report

withKnownIssue_tests_report.md

0.6.09.0 KB
Original Source

Swift Package Manager Tests with withKnownIssue Report

Summary

Found 203 test methods across 37 test files containing withKnownIssue blocks. Below is a detailed breakdown by test suite, platform constraints, and build system impacts.

Major Test Suites with withKnownIssue Tests

1. BuildCommandTestCases

Platform Constraints: Primarily Windows-specific issues Build Systems: All platforms support (Native, SwiftBuild, Xcode)

Key Tests:

  • importOfMissedDepWarning - Build Systems: SwiftBuild, Xcode - Platform: All
  • importOfMissedDepWarningVerifyingErrorFlow - Build Systems: SwiftBuild, Xcode - Platform: All
  • symlink - Build Systems: All - Platform: Windows only
  • buildExistingExecutableProductIsSuccessfull - Build Systems: SwiftBuild only - Platform: Windows only
  • buildExistingLibraryProductIsSuccessfull - Build Systems: SwiftBuild only - Platform: Windows only
  • parseableInterfaces - Build Systems: SwiftBuild only - Platform: Windows only
  • buildSystemDefaultSettings - Build Systems: SwiftBuild only - Platform: Windows only
  • swiftGetVersion - Build Systems: Xcode, SwiftBuild, Native (release) - Platform: Windows or others
  • getTaskAllowEntitlement - Build Systems: SwiftBuild, Xcode - Platform: Non-Linux only

2. PluginTests

Platform Constraints: Heavily Windows-focused issues Build Systems: All platforms support

Key Tests:

  • testUseOfBuildToolPluginTargetByExecutableInSamePackage - Build Systems: All - Platform: Windows only
  • testUseOfBuildToolPluginTargetNoPreBuildCommands - Build Systems: Native (Windows CI), SwiftBuild (all) - Platform: Mixed
  • testLocalAndRemoteToolDependencies - Build Systems: All - Platform: Windows only
  • testIncorrectDependencies - Build Systems: SwiftBuild - Platform: Windows and Linux (CI)
  • testTransitivePluginOnlyDependency - Build Systems: SwiftBuild only - Platform: Windows only

3. TestCommandTests

Platform Constraints: Mixed Windows and general platform issues Build Systems: All platforms support

Key Tests:

  • basicXCTestSupport - Build Systems: All - Platform: Windows only
  • listTests - Build Systems: All - Platform: Windows only
  • testableExecutableTests - Build Systems: All - Platform: General intermittent
  • parallelTests - Build Systems: All - Platform: General intermittent
  • testDiscovery - Build Systems: All - Platform: General intermittent

4. PackageCommandTests

Platform Constraints: General intermittent issues and some Windows-specific Build Systems: All platforms support

Key Tests:

  • archiveSource - Build Systems: All - Platform: General intermittent
  • dumpSymbolGraph - Build Systems: All - Platform: General intermittent
  • Multiple plugin-related tests - Build Systems: All - Platform: General intermittent

5. CFamilyTargetTests

Platform Constraints: General intermittent issues Build Systems: All platforms support

Key Tests:

  • testCLibraryTargets - Build Systems: All - Platform: General intermittent
  • testModuleMapGenerationCases - Build Systems: All - Platform: General intermittent

6. DependencyResolutionTests

Platform Constraints: Windows and general intermittent Build Systems: All platforms support

Key Tests:

  • testInternalSimpleTargets - Build Systems: All - Platform: General intermittent
  • testExternalSimpleTargets - Build Systems: All - Platform: General intermittent
  • testExternalBranchTargets - Build Systems: All - Platform: Windows specific for some

7. ResourcesTests

Platform Constraints: General intermittent issues Build Systems: All platforms support

Key Tests:

  • testBasicResourceAccessibility - Build Systems: All - Platform: General intermittent
  • testResourcesInClangTargets - Build Systems: All - Platform: General intermittent

8. ModuleAliasingFixtureTests

Platform Constraints: General intermittent issues Build Systems: All platforms support

Key Tests:

  • testDirectDeps1 - Build Systems: All - Platform: General intermittent
  • testNestedDeps1 - Build Systems: All - Platform: General intermittent

9. TraitTests

Platform Constraints: Mixed Windows and SwiftBuild issues Build Systems: All platforms support

Key Tests:

  • testBasicTraits - Build Systems: All - Platform: General intermittent
  • testConditionTraits - Build Systems: SwiftBuild, Xcode - Platform: Windows and others

10. TestDiscoveryTests

Platform Constraints: Windows path issues Build Systems: All platforms support

Key Tests:

  • build - Build Systems: All - Platform: Windows only
  • discovery - Build Systems: All - Platform: Windows only
  • asyncMethods - Build Systems: All - Platform: Windows only

Additional Test Suites

CoverageTests

  • codeCoverageBasic - Build Systems: All - Platform: General intermittent
  • codeCoverageFileReport - Build Systems: All - Platform: General intermittent

RunCommandTests

  • swiftRunWithVerbosity - Build Systems: All - Platform: General intermittent
  • swiftRunExecutable - Build Systems: All - Platform: General intermittent

APIDiffTests

  • testAPIDiffOfModuleWithCDependency - Build Systems: All - Platform: General intermittent
  • testAPIDiffOfVendoredCDependency - Build Systems: All - Platform: General intermittent

SwiftSDKCommandTests

  • installationAndUsage - Build Systems: All - Platform: General intermittent

PathTests

Platform Constraints: Windows path handling issues Build Systems: N/A (Basic functionality tests)

Key Tests:

  • Multiple path manipulation tests - Platform: Windows only
  • Path component and validation tests - Platform: Windows only

Platform Distribution Summary

Windows-Specific Issues: ~60% of withKnownIssue tests

  • File path length limitations
  • Platform-specific build failures
  • SwiftBuild integration issues on Windows
  • Path handling and directory operations

General Intermittent Issues: ~35% of tests

  • Network-dependent operations
  • Timing-sensitive operations
  • CI environment variations
  • Build caching issues

Other Platform-Specific: ~5% of tests

  • macOS-only sandboxing tests
  • Linux-specific issues
  • Amazon Linux specific issues

Build System Impact Analysis

SwiftBuild: Most impacted build system

  • Windows platform issues
  • Feature gaps compared to native build system
  • Integration issues with various Swift Package Manager features
  • Plugin system compatibility issues

Native: Generally more stable

  • Occasional issues with newer features
  • Some Windows-specific problems
  • Better plugin support

Xcode: Moderate impact

  • Some feature gaps
  • Integration testing needed for certain scenarios
  • Limited plugin functionality

Key Patterns Observed

  1. Platform-Conditional Issues: Most withKnownIssue blocks use when: clauses to specify platform or build system constraints
  2. Intermittent Failures: Many tests marked with isIntermittent: true for CI stability
  3. Feature Gaps: SwiftBuild often has missing features compared to native build system
  4. Path Issues: Windows long path handling is a recurring theme
  5. Integration Issues: Plugin system, test discovery, and build tool integration show most issues
  6. Build System Parity: Tests often have different expectations per build system
  7. CI Environment Issues: Many failures are environment-specific

Recommendations

  1. Windows Support: Focus on improving Windows compatibility, especially for:

    • Path length handling
    • SwiftBuild integration
    • File system operations

  2. SwiftBuild Parity: Address feature gaps between SwiftBuild and native build system

  3. CI Stability: Investigate and resolve intermittent failures to improve test reliability

  4. Plugin System: Strengthen plugin system integration across all build systems

  5. Test Infrastructure: Consider improving test isolation to reduce platform-specific issues

Report generated from analysis of 203 withKnownIssue test methods across 37 test files in the Swift Package Manager test suite.