Back to Hello Agents

SRE On-Call Agent

Co-creation-projects/zjzhou-SREOnCallAgent/main.ipynb

1.0.39.1 KB
Original Source

SRE On-Call Agent

AI-powered incident triage, root cause investigation, and post-mortem generation

Project Introduction

When a production alert fires at 3am, an SRE on-call engineer must:

  1. Triage — assess severity and plan the investigation
  2. Investigate — search logs, query metrics, consult runbooks
  3. Write a post-mortem — root cause, timeline, action items

This project automates that workflow with a three-agent AI pipeline:

Alert JSON
    │
    ▼
┌─────────────────────────────┐
│  Stage 1: TriageAgent       │  Plan-and-Solve
│  "What should I investigate?"│  → ordered investigation plan
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│  Stage 2: InvestigationAgent│  ReAct loop
│  log_search / metric_query  │  → root cause hypothesis
│  / runbook_lookup           │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│  Stage 3: PostmortemAgent   │  Reflection
│  draft → critique → revise  │  → final RCA report
└─────────────────────────────┘

Author Information

  • GitHub: @zjzhou
  • Date: 2026-04-21
  • HelloAgents Chapter: 16 (Graduation Project)

Part 2: Environment Setup

python
# Install dependencies
!pip install -q openai fastapi uvicorn pyyaml pydantic python-dotenv
python
import os
import sys
import json
import time
import re
import ast
import glob
import yaml
from pathlib import Path
from typing import Dict, Any, List
from dotenv import load_dotenv
from openai import OpenAI

# Add project root to path so src/ imports work
project_root = Path().resolve()
if str(project_root) not in sys.path:
    sys.path.insert(0, str(project_root))

load_dotenv()
print("✅ Environment loaded")
print(f"   LLM_MODEL_ID: {os.getenv('LLM_MODEL_ID', 'NOT SET')}")
print(f"   LLM_BASE_URL: {os.getenv('LLM_BASE_URL', 'NOT SET')}")
print(f"   LLM_API_KEY:  {'SET' if os.getenv('LLM_API_KEY') else 'NOT SET'}")

Part 3: Tool Definitions

Three tools give the InvestigationAgent eyes into the incident:

ToolInputWhat it does
LogSearchToolkeyword/regexSearches incident log entries
MetricQueryToolmetric name keywordReturns time-series metric data
RunbookLookupToolerror patternFetches runbook remediation steps
python
from src.tools.log_search_tool import LogSearchTool
from src.tools.metric_query_tool import MetricQueryTool
from src.tools.runbook_tool import RunbookLookupTool

# --- Quick tool smoke test ---
incident_path = project_root / "data" / "incidents" / "db_pool_exhaustion.json"
with open(incident_path) as f:
    sample_incident = json.load(f)

log_tool = LogSearchTool(sample_incident)
metric_tool = MetricQueryTool(sample_incident)
runbook_tool = RunbookLookupTool(
    service="checkout-service",
    runbooks_dir=str(project_root / "data" / "runbooks")
)

print("🔧 LogSearchTool — searching for 'pool':")
print(log_tool.run("pool"))
print()
print("📊 MetricQueryTool — querying 'db_pool':")
print(metric_tool.run("db_pool"))
print()
print("📖 RunbookLookupTool — pattern 'pool exhausted':")
print(runbook_tool.run("pool exhausted"))

Part 4: Agent Construction

Three agents implementing different paradigms from the Hello-Agents curriculum:

  • TriageAgent — Plan-and-Solve (Chapter 4): LLM creates a structured investigation plan
  • InvestigationAgent — ReAct (Chapter 4): iterative Reason-Act-Observe loop with tools
  • PostmortemAgent — Reflection (Chapter 4): draft → critique → revise
python
from src.core.llm_client import HelloAgentsLLM
from src.agents.triage_agent import TriageAgent
from src.agents.investigation_agent import InvestigationAgent
from src.agents.postmortem_agent import PostmortemAgent
from src.agents.pipeline import run_pipeline, list_incidents, load_incident

# Verify LLM connection
llm = HelloAgentsLLM(verbose=True)
test_response = llm.think([{"role": "user", "content": "Reply with exactly: LLM OK"}])
print(f"\nLLM connection test: {test_response.strip()}")

print(f"\nAvailable incident fixtures: {list_incidents()}")

Part 5: Demo — Full Pipeline

Demo 1: DB Connection Pool Exhaustion (INC-001)

Alert: checkout-service P99 latency 8.3s > threshold 1.0s
Root cause (ground truth): Missing index on orders.user_id → full table scan → DB pool exhaustion

python
# Show the incident alert
incident_1 = load_incident("db_pool_exhaustion")
print("📣 Incident Alert:")
print(json.dumps({
    "incident_id": incident_1["incident_id"],
    "service": incident_1["service"],
    "severity": incident_1["severity"],
    "alert": incident_1["alert"]
}, indent=2))
python
# Run the full pipeline
start = time.time()
result_1 = run_pipeline("db_pool_exhaustion", verbose=True)
elapsed_1 = round(time.time() - start, 1)
print(f"\n⏱️  Pipeline completed in {elapsed_1}s")
python
# Display the generated post-mortem report
from IPython.display import Markdown, display
display(Markdown(result_1["report"]))

Demo 2: External API Rate Limit Cascade (INC-003)

Alert: payment-service success rate 23.4% < threshold 95%
Root cause (ground truth): Promotional traffic spike + no exponential backoff → retry storm → Stripe 429 cascade

python
start = time.time()
result_3 = run_pipeline("external_api_ratelimit", verbose=True)
elapsed_3 = round(time.time() - start, 1)
print(f"\n⏱️  Pipeline completed in {elapsed_3}s")
display(Markdown(result_3["report"]))

Part 6: Performance Evaluation

Run all 3 incident fixtures and measure: pipeline speed, root cause accuracy, and report quality.

python
GROUND_TRUTH = {
    "db_pool_exhaustion":   "missing index",
    "memory_leak_oom":      "no TTL",
    "external_api_ratelimit": "retry storm",
}

eval_results = []
for incident_id in list_incidents():
    t0 = time.time()
    res = run_pipeline(incident_id, verbose=False)
    elapsed = round(time.time() - t0, 1)

    ground_truth_keyword = GROUND_TRUTH.get(incident_id, "")
    root_cause_text = res["findings"].get("root_cause", "").lower()
    correct = ground_truth_keyword.lower() in root_cause_text

    eval_results.append({
        "incident_id": incident_id,
        "service": res["service"],
        "severity": res["severity"],
        "elapsed_s": elapsed,
        "root_cause_correct": "✅" if correct else "❌",
        "root_cause_found": res["findings"].get("root_cause", "")[:80] + "...",
    })

print("\n=== Evaluation Results ===")
for r in eval_results:
    print(f"  [{r['root_cause_correct']}] {r['incident_id']:30s} "
          f"| {r['elapsed_s']:5.1f}s | {r['root_cause_found']}")

accuracy = sum(1 for r in eval_results if r["root_cause_correct"] == "✅") / len(eval_results)
avg_time = sum(r["elapsed_s"] for r in eval_results) / len(eval_results)
print(f"\nAccuracy: {accuracy:.0%}   Avg pipeline time: {avg_time:.1f}s")

Part 7: Summary and Outlook

What We Built

  • TriageAgent (Plan-and-Solve): converts a raw alert into an ordered investigation plan, reducing hallucinated tool calls in the ReAct stage
  • InvestigationAgent (ReAct): iterates through log search → metric queries → runbook lookup to arrive at a root cause with evidence
  • PostmortemAgent (Reflection): produces a structured RCA report and self-critiques it against quality criteria before finalizing
  • FastAPI backend: exposes the pipeline as a REST API, ready for frontend integration

Agent Paradigms Demonstrated

AgentParadigmChapter Reference
TriageAgentPlan-and-SolveChapter 4
InvestigationAgentReActChapter 4
PostmortemAgentReflectionChapter 4

Challenges and Lessons

  1. Structured output parsing: LLMs don't always return valid JSON/Python lists. Robust parsing with fallback logic is essential.
  2. Tool design: Tool descriptions must be precise — ambiguous descriptions cause the ReAct agent to call the wrong tool.
  3. Context window management: Passing the full investigation history in each ReAct turn grows quickly. In production, use a sliding window or summarization.
  4. Reflection threshold: The Reflection pattern only adds value if the critique is rigorous. Vague critique prompts produce vague improvements.

Future Plans

  • SSE streaming: stream agent reasoning steps to a frontend in real-time
  • Vue/React frontend: incident selector UI + live reasoning trace + markdown report viewer
  • Real log ingestion: connect to actual log aggregators (Loki, CloudWatch, Datadog)
  • Memory across incidents: vector store for past RCA reports to accelerate future investigations
  • Automated runbook execution: give the agent permission to run safe remediation commands (kubectl, SQL index creation)