DSL Compiler Design: ANTLR4 + Javassist

Overview

SkyWalking OAP server uses four domain-specific languages (DSLs) for telemetry analysis. All four share the same compilation tech stack: ANTLR4 for grammar parsing and Javassist for runtime bytecode generation.

Compilation Pipeline

All four DSLs follow the same three-phase compilation pipeline at OAP startup:

DSL string (from .oal script or YAML config)
    |
    v
Phase 1: ANTLR4 Parsing
    Lexer + Parser (generated from .g4 grammars at build time)
    → Immutable AST model
    |
    v
Phase 2: Java Source Generation
    Walk AST model, emit Java source code as strings
    |
    v
Phase 3: Javassist Bytecode Generation
    ClassPool.makeClass() → CtClass → addMethod(source) → toClass()
    → Ready-to-use class instance loaded into JVM

What Each DSL Generates

OAL is the most complex -- it generates three classes per metric (metrics class with storage annotations, metrics builder for serialization, and source dispatcher for routing), whereas MAL/LAL/Hierarchy each generate a single functional class per expression.

ANTLR4 Grammars

Each DSL has its own ANTLR4 lexer and parser grammar. The Maven ANTLR4 plugin generates Java lexer/parser classes at build time; these are then used at runtime to parse DSL strings.

Javassist Constraints

Javassist compiles Java source strings into bytecode but has limitations that shape the code generation:

• No anonymous inner classes or lambdas -- Callback-based APIs require workarounds. LAL uses private methods called directly from execute() instead of Consumer callbacks. OAL pre-compiles callbacks as separate CtClass instances where needed.
• No generics in method bodies -- Generated source uses raw types with explicit casts.
• Class loading anchor -- Each DSL uses a PackageHolder marker class so that ctClass.toClass(PackageHolder.class) loads the generated class into the correct module/package (required for JDK 9+ module system).

OAL additionally uses FreeMarker templates to generate method bodies for metrics classes, builders, and dispatchers, since these classes are more complex and benefit from template-driven generation.

Module Structure

oap-server/
  oal-grammar/            # OAL: ANTLR4 grammar
  oal-rt/                 # OAL: compiler + runtime (Javassist + FreeMarker)
  analyzer/
    meter-analyzer/       # MAL: grammar + compiler + runtime
    log-analyzer/         # LAL: grammar + compiler + runtime
    hierarchy/            # Hierarchy: grammar + compiler + runtime
    agent-analyzer/       # Calls MAL compiler for meter data

OAL keeps grammar and runtime in separate modules (oal-grammar and oal-rt) because server-core depends on the grammar while the runtime implementation depends on server-core (avoiding circular dependency). MAL, LAL, and Hierarchy are each self-contained in a single module.

Groovy Replacement (MAL, LAL, Hierarchy)

Reference: Discussion #13716

MAL, LAL, and Hierarchy previously used Groovy as the runtime scripting engine. OAL has always used ANTLR4 + Javassist. The Groovy-based DSLs were replaced for the following reasons:

1. Startup cost -- 1,250+ GroovyShell.parse() calls at OAP boot, each spinning up the full Groovy compiler pipeline.

2. Runtime execution overhead -- MAL expressions execute on every metrics ingestion cycle. Per-expression overhead from dynamic Groovy compounds at scale: property resolution through 4+ layers of indirection, ExpandoMetaClass closure allocation for simple arithmetic, and megamorphic call sites that defeat JIT optimization.

3. Late error detection -- MAL uses dynamic Groovy; typos in metric names or invalid method chains are only discovered when that specific expression runs with real data.

4. Debugging complexity -- Stack traces include Groovy MOP internals (CallSite, MetaClassImpl, ExpandoMetaClass), obscuring the actual expression logic.

5. GraalVM incompatibility -- invokedynamic bootstrapping and ExpandoMetaClass are fundamentally incompatible with ahead-of-time (AOT) compilation, blocking the GraalVM native-image distribution.

The DSL grammar for users remains 100% unchanged -- the same expressions written in YAML config files work exactly as before. Only the internal compilation engine was replaced.

Verification: Groovy v1 Checker

To ensure the new Java compilers produce identical results to the original Groovy implementation, a dual-path comparison test suite is maintained under test/script-cases/:

test/script-cases/
  scripts/
    mal/                      # Copies of shipped MAL configs (test-otel-rules, test-meter-analyzer-config, etc.)
    lal/                      # Copies of shipped LAL scripts (test-lal/)
    hierarchy-rule/           # Copy of shipped hierarchy-definition.yml
  script-runtime-with-groovy/
    mal-v1-with-groovy/       # MAL v1: original Groovy-based implementation
    lal-v1-with-groovy/       # LAL v1: original Groovy-based implementation
    hierarchy-v1-with-groovy/ # Hierarchy v1: original Groovy-based implementation
    mal-lal-v1-v2-checker/    # Runs every MAL/LAL expression through BOTH v1 and v2, compares results
    hierarchy-v1-v2-checker/  # Runs every hierarchy rule through BOTH v1 and v2, compares results

The checker mechanism:

1. Loads all test copies of production YAML config files from test/script-cases/scripts/
2. For each DSL expression, compiles with both v1 (Groovy) and v2 (ANTLR4 + Javassist)
3. Compares the results:
   • MAL: Two-level comparison for each expression:
     1. Metadata comparison -- sample names, aggregation labels, downsampling type, percentile config
     2. Runtime execution comparison -- builds mock SampleFamily input data from ExpressionMetadata, executes with both v1 and v2, compares output samples (count, labels, values with epsilon). For increase()/rate() expressions, the CounterWindow is primed with an initial run before comparing the second run's output.
   • LAL: Runtime execution comparison -- both v1 and v2 execute with mock LogData, then compare execution state (service, layer, tags, abort/save flags). For rules requiring extraLog (e.g., envoy-als), mock proto data is built from .input.data files and the LALSourceTypeProvider SPI resolves the proto type per layer. Test scripts include both copies of production configs (oap-cases/) and dedicated feature-coverage rules (feature-cases/).
   • Hierarchy: Compare BiFunction evaluation with test Service pairs

This ensures 100% behavioral parity. The Groovy v1 modules are test-only dependencies -- they are not included in the OAP distribution.