Dynamic Code Generation and Debugging

SkyWalking OAP server uses four Domain-Specific Languages (DSLs) to define observability logic: OAL (traces/mesh metrics), MAL (meter metrics), LAL (log analysis), and Hierarchy (service matching rules). These DSL scripts are compiled into JVM bytecode when the OAP server starts. The generated classes run in-process — there are no intermediate source files.

When a runtime error occurs inside these generated classes, the stack trace references class names and source locations that map back to the original DSL configuration files. This document explains how to dump the generated bytecode to disk for inspection and how to read the error messages.

DSL Configuration Files

All paths are relative to the OAP distribution root directory.

Dumping Generated Classes

Set the environment variable SW_DYNAMIC_CLASS_ENGINE_DEBUG to any non-empty value before starting the OAP server. All four DSL compilers check this variable and dump .class files to disk when it is set.

# Binary distribution
export SW_DYNAMIC_CLASS_ENGINE_DEBUG=Y
bin/oapService.sh

# Docker
docker run -e SW_DYNAMIC_CLASS_ENGINE_DEBUG=Y ... apache/skywalking-oap-server

# Kubernetes (in container env section)
env:
  - name: SW_DYNAMIC_CLASS_ENGINE_DEBUG
    value: "Y"

Output Directory Structure

The generated .class files are written to sibling directories next to oap-libs/:

Binary distribution (apache-skywalking-apm-bin/):

apache-skywalking-apm-bin/
├── config/                   ← DSL source scripts (*.oal, *.yaml, *.yml)
├── oap-libs/                 ← OAP server jars
├── oal-rt/                   ← Generated OAL classes
│   ├── metrics/              ←   e.g., ServiceRespTimeMetrics.class
│   ├── metrics/builder/      ←   e.g., ServiceRespTimeMetricsBuilder.class
│   └── dispatcher/           ←   e.g., ServiceDispatcher.class
├── mal-rt/                   ← Generated MAL classes (e.g., vm_L25_cpu_total_percentage.class, vm_L20_filter.class)
├── lal-rt/                   ← Generated LAL classes (e.g., default_L3_default.class)
└── hierarchy-rt/             ← Generated Hierarchy classes (e.g., hierarchy_definition_L88_name.class)

Docker (/skywalking/):

/skywalking/
├── config/
├── oap-libs/
├── oal-rt/
├── mal-rt/
├── lal-rt/
└── hierarchy-rt/

The OAL output directories are cleaned on each restart. MAL, LAL, and Hierarchy directories are created on demand if they don't exist.

Inspecting Generated Classes

Use javap to decompile a generated .class file:

javap -v -p oal-rt/metrics/ServiceRespTimeMetrics.class

The output includes:

• SourceFile attribute — shows the DSL source file and the generated class name.
• LineNumberTable — maps bytecode offsets to statement numbers, used by the JVM in stack traces.
• LocalVariableTable — shows named local variables for readability.

Reading Error Stack Traces

When a runtime error occurs inside a generated class, the JVM prints a stack trace that combines the SourceFile attribute and the LineNumberTable. The format is:

at <package>.<ClassName>.<method>(SourceFile:LineNumber)

The SourceFile attribute encodes the original DSL configuration file in parentheses:

(<dsl_source_file>:<rule_line_or_index>)<GeneratedClassName>.java

Example Stack Trace

java.lang.ArithmeticException: / by zero
    at ...metrics.generated.ServiceRespTimeMetrics.id0((core.oal:20)ServiceRespTimeMetrics.java:3)
    at ...worker.MetricsStreamProcessor.in(MetricsStreamProcessor.java:...)
    ...

Reading this:

• (core.oal:20) — the error originates from OAL file core.oal, line 20
• ServiceRespTimeMetrics.java — the generated class for metric ServiceRespTime
• :3 — statement 3 within the generated id0 method

Format Per DSL

Notes:

• The class name pattern is {yamlFileName}_L{lineNo}_{ruleName} for all DSLs (except OAL). The yaml file name and line number from yamlSource are combined with the rule name or filter.
• The number after : in the SourceFile prefix is the line number in the YAML file where the rule is defined (for MAL in production, this may be a 0-based rule index instead of a line number).
• When source information is unavailable, the class name falls back to MalExpr_<N> / LalExpr_<N> / HierarchyRule_<N> and the SourceFile to just ClassName.java without the parenthesized prefix.

Mapping Back to DSL Source

1. Identify the DSL type from the package or class name:

   • ...metrics.generated.*Metrics or ...Dispatcher → OAL
   • ...meter.analyzer.v2.compiler.rt.* → MAL (class name: {yamlName}_L{lineNo}_{ruleName} or MalExpr_<N>)
   • ...log.analyzer.v2.compiler.rt.* → LAL (class name: {yamlName}_L{lineNo}_{ruleName} or LalExpr_<N>)
   • ...hierarchy.rule.rt.* → Hierarchy (class name: {yamlName}_L{lineNo}_{ruleName} or HierarchyRule_<N>)

2. Find the source file from the parenthesized prefix in the SourceFile attribute (e.g., core.oal, vm.yaml), or from the class name prefix (e.g., vm_L25_... → vm.yaml). These files are in the config/ directory.

3. Locate the rule using the line number from the class name (_L25_) or the SourceFile prefix (:25).

4. Use the statement number (after the last :) as a rough indicator of which operation within the generated method failed. Dump the class (see above) and use javap -v to see the exact mapping.

Generating All DSL Classes Offline

You can compile every DSL script from the source tree without starting the OAP server. This is useful for verifying that all scripts are syntactically valid after editing, or for batch-inspecting the generated bytecode with javap or an IDE decompiler.

The DSLClassGeneratorTest in the server-starter module compiles all OAL, MAL, LAL, and Hierarchy scripts and dumps the .class files to target/generated-dsl-classes/.

# From the project root (requires a prior build: ./mvnw -Pbackend install -Dmaven.test.skip)
./mvnw test -pl oap-server/server-starter \
  -Dtest="DSLClassGeneratorTest#generateAllDSLClasses" \
  -Dcheckstyle.skip

Output

oap-server/server-starter/target/generated-dsl-classes/
├── oal/              ← Metrics, MetricsBuilder, Dispatcher classes
├── mal/              ← MAL expression and filter classes
├── lal/              ← LAL expression classes
└── hierarchy/        ← Hierarchy rule classes

The test fails if any script cannot be compiled and prints the list of failures.

What It Covers

Common Error Patterns

OAL Compilation Failure

OAL compilation errors are logged at ERROR level during OAP startup:

ERROR o.a.s.o.v.g.OALClassGeneratorV2 - Can't generate method id for ServiceRespTimeMetrics.

This indicates that the generated Java source for the id method failed to compile. Check the OAL script syntax at the reported metric name.

MAL/LAL Runtime Error

MAL and LAL errors during metric processing are caught and logged per-expression:

ERROR o.a.s.o.m.a.v.MetricConvert - Analyze Analyzer{...} error
java.lang.NullPointerException
    at ...vm_L25_cpu_total_percentage.run((vm.yaml:25)cpu_total_percentage.java:5)

This tells you: the error is in vm.yaml, line 25, metric cpu_total_percentage, at statement 5 of the generated run() method. The processing continues for other metrics — a single expression failure does not crash the server.

Hierarchy Compilation Failure

Hierarchy rule compilation errors are thrown at startup:

IllegalStateException: Failed to compile hierarchy rule: lower-short-name-remove-namespace,
    expression: { (u, l) -> { if (...) { ... } } }

Check the rule expression syntax in config/hierarchy-definition.yml under auto-matching-rules.