Checkstyle's testing techniques

The video walkthrough of this page with an example

Test Driven Development (TDD)

TDD is one of the encouraged ways to efficiently make new implementations for Checkstyle.

With TDD, you create the test prior to the implementation that it will test. The created test then serves as a door to the module's source code, which we can access by using the debug tool to reproduce its execution piece by piece.

TDD in turn provides a precise experiment field for bringing a new implementation to the module, allowing you to develop with respect to that test's results.

Where Checkstyle's tests are located

The directory with tests that we need is located in src/test/java/com/puppycrawl/tools/checkstyle/,,,. What follows after is the relative path of the module that the needed tests belong to.

• Tests are organized into module data collection files. Each such collection file stores all of the tests for a certain module
• The Name format for such collection files is: [ModuleClassName]Test where [...] indicates you paste your own option inside

Test structure

Each method with a test prefix in module data collection files represents a distinct test. Such test methods come in the following common structure:

@Test
public void test[NameOfTest]() throws Exception {
  final String[] expected = {
          ...
  }
  verifyWithInlineConfigParser(
          getPath("InputFile.java"), expected);
}

Let's go over each essential part of the test method structure

1. Input file

Input files are files that are being tested by the test.

All input files are located in the same src/test directory, but in resources or resources-noncompilable folders.

• At this moment, Checkstyle uses jdk 21 as its operating version, and all input files that can be compiled with jdk 21 or older go to resources, while those input files that require newer versions of jdk to compile go to resources-noncompilable.

Input files are named in the following format: Input[ModuleName][FileNickname].java

• Ex. The input file name with a nickname of "Class" for the AnnotationLocation check would be: InputAnnotationLocationClass.java
• Although you can put anything for [FileNickname], it is encouraged that the nickname is the same as, or at least has resemblance to the test's name it belongs to.

2. Expected violations

• The String array, String[] expected, contains the violation messages that we expect to receive from the test.
• These are not always the violation messages that are guaranteed to come out of the test, but rather the violation messages that the test should give if we assume that all necessary implementation for that test's module is correctly in effect.
• The format for an expected violation message element in the expected array is:
  "lineNumber:columnNumber: " + <rest of violation message that can be generated using getCheckMessage(messageCode)>

3. Verification

Once the input file and expected violations are properly connected to the test, it's time for the verify method (usually verifyWithInlineConfigParser()) to attempt to use Checkstyle to analyze the input file, get the actual violations that it detects with the current implementation, and at the end compare actual results with the expected. If actual results differ from the expected, the test fails, and explains where exactly the inconsistency was.

Input file structure and Behavior Driven Development

Checkstyle input files have the following abstract structure:

/*
...
config
...
*/

package [inputPathPackage]
        ...

public class InputFileName {
  ...  
  ...
  ... // violation, '...'
  ...
}

This structure manifests the concept of Behavior Driven Development (BDD). BDD simulates the results of before, during, and after states of execution in an illustrative way. This makes it easier for developers to access and be aware of the source's status at each phase of execution.

For input files, these states are implemented in 3 parts

Config (Before)

Enclosed in block comments /* ... */ at the beginning of the file, the config represents the before state of the input file as it shows the settings used for the module to run on the code.

There are various types/formats of configs such as property, XML, and java. However, in regular tests, the property format is required to be used at all times, unless there is an exception such as when a module has no properties (because the property config requires at least some property to be defined), then the XML format is used. The Java config format is the last resort if others don't help, being defined outside the input file in the test method and thus violating the BDD.

The property config format has the following layout:

/*
ModuleName
propertyName = propertyValue
otherPropertyName = (default)otherPropertyDefaultValue
...
lastPropertyName = lastPropertyValue

*/

Note: Although not shown here, it is encouraged to leave 2 blank lines at the end of the config before closing comment block to avoid having to change subsequent violation lines when new property is added.

Tested Code (During)

Next after the config comes the rest of the Java source code that serves as the during state of input file's BDD. This is the intended piece to be tested and to be covered by the running test's scope.

Violation marking (After)

The final part represents the after state of input file by showing the intended results of the test using specific // violation comment marking on the same lines of code that intend to cause the violation.

The format of the violation comment is: // violation, 'violation message' where violation message has to belong to any of the module's existing violation message options, but not necessarily be entirely written out between the quotes but at least just some part.

Conclusion

Once you have the test set up properly, you now have a handy tool for accurately analyzing the module by using the test to access the module's source in real time.

• To do so, place a breakpoint anywhere in the module's source and run the test method with debug. This will get you to that point of module's source in debug mode.

This Test Driven Development technique with collaboration with the debug fits well for solving various module problems from corresponding issues that give you already prepared information for creating the input file and the test to solve that problem.

To watch the live moment on applying TDD to solve module's problem, you can take a look at the 2nd part of video tutorial mentioned at the beginning of this page.