Deep dive

Everything beyond the Quick start for the msb-metrics sidecar: the complete flag set, what it emits, the attributes attached to each datapoint, and the operational notes you'll want once it's running in earnest.

Deployment constraints

msb-metrics reads the shm registry directly. Two constraints follow.

Same Unix user as msb

The shm object is mode 0600 (owner read/write only). Running msb-metrics as a different user produces EACCES on attach.

Same $MSB_HOME

The shm name is derived from stable_hash($MSB_HOME), so both processes must agree on it. Pass --msb-home explicitly if your environment doesn't set $MSB_HOME; the default is ~/.microsandbox.

Per-host

The registry is per-host. One msb-metrics process per host covers every running sandbox there.

Metrics emitted

All metrics are emitted under the microsandbox.* namespace so they don't collide with OTel semantic-convention system.* host metrics in the same backend tenant. The table below shows the suffix only; the fully-qualified name is microsandbox.<suffix>.

Cumulative byte fields are emitted as gauges carrying the absolute cumulative value. Use rate() (PromQL, OTel-flavored Prom) for throughput. The reason: each shm snapshot already carries an absolute value, and counter add() semantics would require us to track per-sandbox deltas across runs.

The collector also emits its own operational series; see Collector self-observability below.

Collector self-observability

msb-metrics otel ships its own operational metrics through the same OTLP pipeline as the per-sandbox series, so a user can confirm the sidecar is actually flowing using the same Prometheus / Grafana / Datadog queries the rest of their telemetry runs through:

Scope: these series share the same OTel scope as the sandbox metrics, so they show up under otel_scope_name="microsandbox-metrics-collector" with otel_scope_version=<msb version>.

A few queries you'll want to wire up:

```bash
rate(microsandbox_collector_exports_success_total[1m])
```

```bash
rate(microsandbox_collector_exports_failure_total[5m])
  /
clamp_min(
  rate(microsandbox_collector_exports_success_total[5m]) +
  rate(microsandbox_collector_exports_failure_total[5m]),
  1
)
```

```bash
time() - microsandbox_collector_last_success_timestamp_seconds > 300
```

Attributes

Every datapoint carries a configurable set of attributes.

Resource attributes describe the source. Defaults are set automatically; --resource KEY=VALUE overrides or adds.

Identity attributes describe which sandbox a datapoint belongs to. run_id and pid are opt-in because they create a fresh time series per sandbox restart, which inflates active-series counts on cardinality-billed backends.

All flags

msb-metrics stdout [--collect-interval=<dur>]
                   [--flush-interval=<dur>]
                   [--max-buffered=<n>]
                   [--export-timeout=<dur>]
                   [--msb-home=<path>]

The output format is not a stable contract; don't pipe it into production parsers. Sample line:

2026-05-30T02:44:31Z sandbox=devbox id=33 cpu=0.000107 \
    mem=13.6 MiB / 512.0 MiB disk_r=89.3 MiB disk_w=644.7 MiB \
    net_rx=48.0 MiB net_tx=268.5 KiB uptime=2475m15s

Or just run msb-metrics otel --help for the full prose.

Tuning at scale

At ~1000 sandboxes per host the per-exporter buffer dominates heap usage. The shm registry stays a fixed ~512 KiB regardless of count, and the hot path is pure shm (no sqlite read).

Worst-case heap is --max-buffered × active sandboxes × ~350B, reached only when the backend is slow enough to fill the buffer.

Shutdown behavior

SIGINT or SIGTERM triggers a clean drain:

1. Stop the collect ticker.
2. Push any buffered collections through one final export.
3. Call each exporter's shutdown() (OTel: flushes and closes the OTLP transport).
4. Exit.

If an exporter's final export hangs, it's bounded by --export-timeout.

Backend unreachable

Failed exports are retried on the next flush; the failed batch is restored to the front of the buffer. If failures keep arriving, oldest collections drop first and the next successful export's droppedCollectionCount reports how many were lost (and increments the microsandbox.collector.collections.dropped counter). The collector itself does not crash.

The worker uses capped exponential backoff between scheduled retries: flush_interval, then 2× flush_interval, 4×, up to a 32× cap. At the default 10s flush interval that's a worst case of ~5 minutes between retries during a sustained outage, instead of hammering the backend every 10s. Explicit RunningCollector::flush() calls bypass the backoff gate, so a caller that knows the upstream has recovered can force-retry immediately. On the first successful export the worker logs metrics exporter recovered at INFO and the multiplier resets to 1×.

Stopped sandboxes

A sandbox that stops releases its shm slot. msb-metrics reads the active snapshot only, so a stopped sandbox simply stops appearing in the export stream. Downstream the series goes stale (no fresh datapoints), which is the standard "host gone" signal in Prometheus and most TSDBs. There is no explicit "stopped" event.

Counter resets across sandbox restarts

Disk and network byte fields are cumulative from the sandbox process's point of view. When a sandbox restarts, the runtime gets a fresh slot and the counters start from zero again. rate() is robust to this (it detects counter resets), but in the brief window spanning the restart a query may return a small negative interval before the next sample lands. This is normal counter-reset behavior, not a bug.

Troubleshooting

See also

• Metrics collector: overview and quick start for msb-metrics.
• Metrics backends: end-to-end recipes per provider.