Netdata Log Viewer Plugin

A Netdata external plugin for querying and visualizing systemd journal entries with histogram analysis and faceted search.

Overview

This plugin provides a systemd-journal function that Netdata can call to query journal entries, compute histograms, and return faceted data for visualization in the Netdata dashboard.

Architecture

┌─────────────┐
│   Netdata   │
│   Agent     │
└──────┬──────┘
       │ stdin/stdout
       │ (plugin protocol)
       ↓
┌──────────────────────────┐
│  otel-signal-viewer-plugin   │
│                          │
│  ┌───────────────┐       │
│  │ Journal       │       │
│  │ Handler       │       │
│  └───────────────┘       │
│         ↓                │
│  ┌───────────────┐       │
│  │ Shared State  │       │
│  │ (AppState)    │       │
│  └───────────────┘       │
│         ↓                │
│  ┌───────────────┐       │
│  │ histogram-    │       │
│  │ service       │       │
│  └───────────────┘       │
│         ↓                │
│  ┌───────────────┐       │
│  │ journal       │       │
│  │ (indexing)    │       │
│  └───────────────┘       │
└──────────────────────────┘
         ↓
    ┌─────────┐
    │ Jaeger  │ (tracing)
    └─────────┘

Key Features

• Fast histogram computation using pre-built indexes
• Faceted search across journal fields (PRIORITY, HOSTNAME, etc.)
• Caching with memory + disk tiers for performance
• Distributed tracing via OpenTelemetry/Jaeger
• Metrics tracking for function call success/failure rates

Quick Start

Prerequisites

1. Jaeger (optional, for development visibility):

# NOTE: Port 4318 avoids conflict with Netdata's otel-plugin on 4317
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4317 \
  jaegertracing/all-in-one:latest

Building

cargo build --bin otel-signal-viewer-plugin --release

Running

The plugin is designed to be spawned by Netdata:

# Netdata spawns the plugin automatically when configured
# Configure in /etc/netdata/netdata.conf:

[plugins]
    otel-signal-viewer-plugin = yes

For development/testing:

# Set log level
export RUST_LOG="debug"

# Run Netdata in foreground
sudo netdata -D

# View traces
open http://localhost:16686

Development

See QUICKSTART.md for the fast development loop.

See DEVELOPMENT.md for comprehensive documentation.

Development Workflow

1. Make changes to the plugin code
2. Rebuild (fast, seconds): cargo build --bin otel-signal-viewer-plugin
3. Restart Netdata: sudo systemctl restart netdata
4. View traces in Jaeger: http://localhost:16686
5. Check logs: sudo journalctl -u netdata -f

Key Benefits of Current Architecture

✅ Simple - Single binary, single mode (production-only) ✅ Fast iteration - Rebuild only the plugin, not all of Netdata ✅ Observable - Rich tracing and logging via Jaeger + stderr ✅ Production parity - Develop with exact production setup ✅ No mock infrastructure - No TCP bridges or test harnesses needed

Project Structure

netdata-log-viewer/
├── histogram-service/       # Core business logic (library)
├── otel-signal-viewer-plugin/   # Netdata plugin (binary)
├── types/                   # Shared request/response types
├── watcher-plugin/          # DEPRECATED - no longer needed
├── lv/                      # DEPRECATED - no longer needed
├── DEVELOPMENT.md           # Detailed development guide
├── QUICKSTART.md            # Fast reference guide
└── README.md                # This file

Configuration

The plugin is configured at compile time with sensible defaults:

• Journal path: /var/log/journal
• Cache directory: /mnt/ramfs/foyer-storage
• Memory cache: 10,000 entries
• Disk cache: 64 MiB

To customize, edit create_shared_state() in otel-signal-viewer-plugin/src/main.rs.

Observability

Tracing (Jaeger)

View request traces at http://localhost:16686:

• Function call timelines
• Histogram computation duration
• Lock acquisition times
• Error traces

Logging (Stderr)

Control log verbosity with RUST_LOG:

# Debug everything
export RUST_LOG="debug"

# Selective logging
export RUST_LOG="otel_signal_viewer=trace,histogram_service=debug,journal=info"

Metrics (Netdata)

The plugin reports its own metrics:

• otel_signal_viewer.journal_calls - Successful/failed/cancelled function calls

Function Interface

The plugin exposes the systemd-journal function:

Request:

{
  "after": 1699000000,
  "before": 1699100000,
  "selections": {
    "PRIORITY": ["3", "4"],
    "_HOSTNAME": ["server1"]
  }
}

Response:

{
  "status": 200,
  "facets": [...],
  "histogram": [...],
  "available_histograms": [...],
  "columns": {...},
  "data": [...]
}

Dependencies

Core

• rt - Netdata plugin runtime
• histogram-service - Histogram computation
• journal - Journal file indexing

Tracing

• tracing - Structured logging
• opentelemetry - Distributed tracing
• opentelemetry-otlp - OTLP exporter for Jaeger

Async Runtime

• tokio - Async runtime

Performance

• Index caching: Avoids re-reading journal files
• Parallel processing: Uses Rayon for CPU-bound work
• Lock-free where possible: RwLock allows concurrent reads
• Efficient filtering: Pre-built indexes for fast queries

Troubleshooting

Plugin not starting

# Check logs
sudo tail -f /var/log/netdata/error.log

# Run manually
sudo -u netdata /path/to/otel-signal-viewer-plugin

No traces in Jaeger

# Verify Jaeger is running
docker ps | grep jaeger

# Check connectivity
nc -zv localhost 4317

Slow queries

Check Jaeger traces for:

• Lock contention on shared state
• Cache misses in IndexCache
• Large time ranges

License

[Your license here]

Contributing

[Contributing guidelines here]