Significant Events Evaluations

Evaluations for Significant Events, which assess the quality of LLM-based Knowledge Indicator (KI) feature extraction, KI query generation, KI feature exclusion, and KI feature deduplication across failure scenarios. These evaluations support both qualitative (LLM-as-a-judge + deterministic CODE evaluators) and quantitative (trace-based) metrics.

For general information about writing evaluation tests, configuration, and usage, see the main @kbn/evals documentation.

Evaluation suites

Prerequisites

Snapshot data

Evaluations replay Elasticsearch snapshots from a GCS bucket (significant-events-datasets). The bucket is structured as:

significant-events-datasets/
  <run-id>/
    <dataset>/
      <scenario-snapshot>

Set GCS_CREDENTIALS before starting Scout so Elasticsearch can access the GCS repository:

export GCS_CREDENTIALS='{"type":"service_account",...}'

The default run ID is pinned in code (SIGEVENTS_SNAPSHOT_RUN). Override it at runtime:

SIGEVENTS_SNAPSHOT_RUN=2026-02-25 node scripts/evals run --suite significant-events --judge gemini-3-pro

Tracing setup (optional — for token and latency metrics)

To capture trace-based metrics (input/output/cached tokens, and latency), configure tracing exporters and run the EDOT Collector.

Step 1: Configure tracing exporters

Add the following to kibana.dev.yml:

elastic.apm.active: false
elastic.apm.contextPropagationOnly: false
telemetry.enabled: true
telemetry.tracing.enabled: true
telemetry.tracing.sample_rate: 1
telemetry.tracing.exporters:
  - http:
      url: 'http://localhost:4318/v1/traces'

Note: elastic.apm.active: false and elastic.apm.contextPropagationOnly: false are required — Elastic APM and OpenTelemetry tracing cannot run simultaneously. The Scout evals_tracing config set handles this automatically, but when configuring kibana.dev.yml directly you must set both.

Optionally include the Phoenix exporter for a trace UI:

telemetry.tracing.exporters:
  - phoenix:
      base_url: 'https://<my-phoenix-host>'
      public_url: 'https://<my-phoenix-host>'
      project_name: '<my-name>'
      api_key: '<my-api-key>'
  - http:
      url: 'http://localhost:4318/v1/traces'

Step 2: Start EDOT Collector

Ensure Docker is running, then start the EDOT Gateway Collector:

node scripts/edot_collector.js

Or point it at a separate trace cluster:

ELASTICSEARCH_HOST=https://<username>:<password>@<trace-cluster-url> node scripts/edot_collector.js

Without tracing infrastructure, token and latency evaluators gracefully return score: null — all other evaluators are unaffected.

Running evaluations

Start Scout server

node scripts/scout.js start-server --arch stateful --domain classic --serverConfigSet evals_tracing

Run all evaluations

Note: Use Gemini 3 Pro as the evaluation judge to ensure consistent scoring across models. This keeps LLM-as-a-judge criteria evaluations comparable regardless of which model is being evaluated.

node scripts/evals run \
  --suite significant-events \
  --project <connector-id> \
  --judge <gemini-3-pro-connector-id>

Run a specific dataset

SIGEVENTS_DATASET=otel-demo node scripts/evals run \
  --suite significant-events \
  --project <connector-id> \
  --judge <gemini-3-pro-connector-id>

Run a specific spec file

node scripts/evals run \
  --suite significant-events \
  --project <connector-id> \
  --judge <gemini-3-pro-connector-id> \
  ki_feature_extraction.spec.ts

CLI options

Environment variables

Collected metrics

Deterministic (CODE) evaluators

LLM-as-a-judge evaluators

Trace-based evaluators

Adding a new dataset

1. Create a capture script

Each dataset should have its own capture script under scripts/ so that snapshot creation is fully reproducible. See capture_otel_demo_snapshots.ts as the reference implementation.

A capture script typically:

1. Connects to Elasticsearch and Kibana (via getConnectionConfig)
2. Registers a GCS snapshot repository (via registerGcsRepository)
3. Generates or ingests log data into logs* (dataset-specific — e.g. deploy an app, run synthtrace, replay from an external source)
4. Enables Streams and triggers KI feature extraction (via the shared significant_events_workflow helpers)
5. Snapshots logs* and extracted KIs to GCS (via createSnapshot)
6. Cleans up between scenarios

The shared helpers in scripts/lib/ handle GCS registration, snapshot creation, KI feature extraction orchestration, and ES/Kibana connection — the capture script only needs to provide the data generation logic specific to its dataset.

Register the script entry point in scripts/ (e.g. scripts/capture_sigevents_my_app_snapshots.js) so it can be run with:

node scripts/capture_sigevents_my_app_snapshots.js --connector-id <id> --run-id <run-id>

2. Define the dataset and evaluation criteria

1. Create a dataset file in src/datasets/ (e.g. my_app.ts, following the otel_demo.ts pattern)
2. Define scenarios with evaluation criteria
3. Register the dataset in src/datasets/index.ts

3. Run evals

SIGEVENTS_DATASET=my-app node scripts/evals run --suite significant-events --judge <gemini-3-pro-connector-id>

Adding a new eval spec

1. Create a spec file under evals/ (e.g. my_eval/my_eval.spec.ts)
2. Add evaluators in src/evaluators/ (e.g. my_eval_evaluators.ts) — code evaluators for deterministic checks and LLM-as-a-judge evaluators for qualitative criteria. See the existing files for reference (e.g. ki_feature_extraction/evaluators.ts)
3. If the dataset defines evaluation criteria per scenario, you can reuse the scenario_criteria evaluator - it automatically scores LLM output against the dataset's criteria using an LLM judge, so you don't need to write custom LLM evaluators for each spec
4. Wire up the spec with the dataset scenarios and evaluators