Testing rules

The analyzer_testing package provides an API for testing analysis rules. Tests can be written concisely, encouraging the plugin author to write test cases with good coverage of possible Dart syntax, and the analysis rules themselves.

The test class

Analysis rule tests that are written with the analyzer_testing package's support use a class hierarchy to specify shared variables, helper methods, and set-up and tear-down code. This is all based on the test_reflective_loader package. Here is the basic structure:

import 'package:analyzer_testing/analysis_rule/analysis_rule.dart';
import 'package:my_rule/src/rules/my_rule.dart';
import 'package:test_reflective_loader/test_reflective_loader.dart';

@reflectiveTest
class MyRuleTest extends AnalysisRuleTest {
  @override
  void setUp() {
    rule = MyRule();
    super.setUp();
  }

  // Test cases go here.
}

This test file can be written anywhere in the test directory of the plugin package, maybe at test/my_rule_test.dart.

In this code, we are testing the my_rule analysis rule built in writing rules, which reports any time an 'await expression' is found.

This structure is different from the classic test structure used when writing tests with the test package, in which all tests are declared in anonymous closures passed to the group and test functions. Let's examine the components of the MyRuleTest class.

• class MyRuleTest extends AnalysisRuleTest - The test class uses AnalysisRuleTest, from the analyzer_testing package, as a base. AnalysisRuleTest provides common functionality like assertDiagnostics and newFile.
• void setUp - Override this method to provide some set-up code that is executed before each test. This method must call super.setUp(). This method is where we instantiate the analysis rule that we are testing: rule = MyRule();.

The test cases

The individual test cases are declared as instance methods of this class. Each method whose name starts with test_ is registered as a test case. See the test_reflective_loader package's documentation for more details.

@reflectiveTest
class MyRuleTest extends AnalysisRuleTest {
  // ...

  void test_has_await() async {
    await assertDiagnostics(
      r'''
void f(Future<int> p) async {
  await p;
}
''',
      [lint(33, 5)],
    );
  }

  void test_no_await() async {
    await assertNoDiagnostics(
      r'''
void f(Future<int> p) async {
  // No await.
}
''');
  }
}

Let's look at the APIs used in these test cases:

• assertDiagnostics - This is the primary assertion method used in analysis rule tests. It allows us to assert which diagnostics are reported, for some given Dart source code. The first argument is the source code, and the second is a list of expected diagnostics, ExpectedDiagnostic objects. Generally, ExpectedDiagnostic objects are not manually constructed. Instead, we use the lint() function:
• lint(33, 5) - This utilitiy creates an expected diagnostic object representing the analysis rule specified by the analysisRule getter, which is expected at offset 33, for a length of 5 characters.
• assertNoDiagnostics - This is a convenience utility that asserts that no diagnostcs are reported for the given source code.

Most test cases can be written as simply as the two above, with a single call to assertDiagnostics or assertNoDiagnostics.

Some test cases might involve code with compile-time errors, or warnings. (For example, you might want to verify that the analysis rule does not report when certain error conditions are present, so that the user can focus on fixing the error conditions, and not on spurious lint diagnostics.) Here is an example:

  void test_has_await_in_non_async() async {
    await assertDiagnostics(
      r'''
void f(Future<int> p) {
  await p;
}
''',
      [
        // No lint is reported with this error.
        error(CompileTimeError.UNDEFINED_IDENTIFIER_AWAIT, 27, 5),
      ],
    );
  }

In this example, we assert that the only diagnostic reported for this code is an CompileTimeError.UNDEFINED_IDENTIFIER_AWAIT error.

The entrypoint

All of the test code above comes together when we register the test class in the test file's main function:

void main() {
  defineReflectiveSuite(() {
    defineReflectiveTests(MyRuleTest);
  });
}

With this main function, tests can be run in the same way as class test package tests. They can be run in the usual ways, such as using the IDE, or by running dart test or dart --enable-asserts test/my_rule_test.dart.

Writing stub package sources

Often an analysis rule needs to understand if a type or an element (like a class or a method) is a specific type/element from a specific library in a package. For example, a rule might be concerned with the use of the test function declared in the test_core package. In order to write tests for such a rule, the test code needs to import something like 'package:test_core/test_core.dart'. In order to make such an import meaningful, some stub code needs to be written so that, in fact, a test function is made available by that import.

The AnalysisRuleTest class offers a newPackage method which supports writing code in other packages. newPackage returns a PackageBuilder, which is used to add individual library sources via its addFile method. For example, to write the sources for a stub test function in a package named test_core, you can:

class MyRuleTest extends AnalysisRuleTest {
  @override
  void setUp() {
    newPackage('test_core')..addFile('lib/test_core.dart', r'''
void test(
  Object? description,
  dynamic body(), {
  String? testOn,
  Object? /*Timeout?*/ timeout,
  Object? skip,
  Object? tags,
  Map<String, dynamic>? onPlatform,
  int? retry,
  Object? /*TestLocation?*/ location,
  bool solo = false,
}) {}
''');
    super.setUp();
  }
}

Here are a few tips for writing stub package sources:

• newPackage needs to be called in setUp, before the call to super.setUp.
• For the static analysis purposes of testing analysis rules, it is unnecessary to include function bodies (see the empty test body above).
• It is often not necessary to include all of the types which are needed to write a type or an element, like a function signature (see the location parameter above, which is typed as an Object? instead of a TestLocation?). This can greatly simplify the stubs.