MCPProxy · Engineering decision report

Request queueing & per-upstream concurrency limits

Issue #955 — request queueing / concurrency limit for multi-user deployments · 2026-08-07 · 21-agent research workflow; choke-point analysis verified against the code

Recommendation

Option A — a two-tier semaphore limiter inside managed.Client.CallTool , per-server first, then global.

It is the only placement that covers every in-daemon upstream call path — the audit proved code_execution and activity replay bypass Manager.CallTool and hit the managed client directly. It needs no new dependency (golang.org/x/sync is already pinned), inherits FIFO fairness and ctx-aware queue_timeout from semaphore.Weighted, sidesteps the Manager.CallTool RLock stall hazard, and gets hot-reload propagation free from the existing SetGlobalConfig fan-out. Config: max_concurrent_requests, queue_size, queue_timeout — global + per-server overrides, 0 = unlimited (pure opt-in, zero behavior change on upgrade).

01The choke point (verified)

All four dispatch paths converge on one function — and two of them bypass the Manager entirely, which rules out any limiter placed above the managed client:

call_tool_read|write|destructivemcp.go:2112──┐
legacy call_toolmcp.go:2536──┼── Manager.CallTool ──┐
direct-routing modemcp\_routing.go:216──┘manager.go:1123│
REST /api/v1/tools/call → CallToolDirectmcp.go:5356──(same variants)──┤
                                                                          ├──▶managed.Client.CallToolcode_executionmcp\_code\_execution.go:453-464──── GetClient ─────────┤client.go:640 — limiter hereactivity replayruntime.go:1218-1228──── GetClient ─────────┘

• No limit exists today. The only serialization is sseRequestMu for SSE transports. Stdio upstreams are fully multiplexed by mcp-go v0.57.0 (frame write under stdinMu, responses matched by JSON-RPC id) — so queued goroutines block naturally before the frame write, and a semaphore wrapping CallTool works for stdio too. core/client.go:70,396-405 · mcp-go stdio.go:423-489
• Queue wait won’t eat call_tool_timeout: that 2-minute context is created deeper, inside core.Client.CallTool — acquiring before delegation keeps queue time separate. core/client.go:430-445
• Server edition is covered automatically: the multi-user Router only filters visibility; all users share one managed client per server, so per-server limits bound aggregate multi-user load with zero extra code. internal/serveredition/multiuser/router.go
• Deliberately outside the limit: retrieve_tools (local Bleve, no upstream traffic), ListTools (already leader/follower-coalesced), health-check Ping (5s lightweight probe), and the separate-process CLI debug client. managed/client.go:504-637, 971-985 · cli/client.go:266
• Hot-reload is free: per-server config flows through atomic GetConfig/SetConfig; global config fans out via Manager.SetGlobalConfig. Caveat: semaphore.Weighted can’t resize, so reload means an atomic limiter swap with release-closures bound to the old instance. manager.go:298-314

02Options

New internal/upstream/limiter package on x/sync/semaphore. Each limiter = two weighted semaphores: an admission semaphore sized max_concurrent + queue_size acquired with TryAcquire (failure = queue full → instant shed, Envoy max_pending_requests semantics), and a run semaphore sized max_concurrent acquired ctx-aware under queue_timeout. One limiter per server (registry keyed by name) + one global; acquired per-server-first inside managed.Client.CallTool.

Pros

• Covers 100% of in-daemon upstream traffic including the code_execution and replay bypass paths — the only placement that does.
• No new dependency; semaphore.Weighted gives FIFO fairness, cancellable waits, and queue_timeout free.
• Server edition works automatically (shared managed client per server bounds aggregate multi-user load).
• Hot-reload free via existing SetGlobalConfig fan-out + per-server config atomics.
• Avoids the Manager.CallTool RLock hazard (blocking there stalls AddServer/RemoveServer writers).
• Registry keyed by server name is forward-compatible with Spec 074 per-(user,server) brokered pools.
• Protects stdio naturally — the transport genuinely multiplexes, so waiters just block before the frame write.

Cons

• Touches a hot, subtle file (managed/client.go) — must not regress the state machine, ListTools coalescing, or reconnect paths.
• Typed shed errors must propagate to mcp.go/httpapi for correct rendering (isError vs 429) — a small cross-layer contract.
• Semaphores can’t resize → hot-reload needs the atomic-swap-with-bound-release-closures pattern (fiddly but unit-testable).
• Doesn’t govern the separate-process CLI debug client — acceptable; document it.

Global limit in the mcp.go call_tool dispatch; per-server limit in Manager.CallTool. No managed-client changes.

Pros

• Smaller upstream-layer diff; mcp.go already has activity/span/metrics seams in place.
• Global sheds map cleanly to MCP tool errors with no error-type plumbing.

Cons

• Verified hole: code_execution and replay call the managed client directly — one script can still stampede a stdio upstream, defeating the issue’s core requirement.
• Manager.CallTool holds m.mu.RLock for the whole call; a limiter blocking under it stalls AddServer/RemoveServer — fixing that replicates the drop-and-reacquire dance Option A avoids.
• Two enforcement sites; future call paths must remember to route through them.

A chi middleware capping concurrent inbound requests on /mcp and /api/v1, returning 429 when saturated. No per-upstream awareness.

Pros

• Trivial; standard reverse-proxy pattern; clean 429 semantics on REST; protects the daemon’s own goroutine budget.

Cons

• Cannot express per-upstream limits — one slow stdio server still absorbs unbounded calls while fast servers get throttled: the opposite of what #955 asks.
• Amplification blind spot: one code_execution request fans out to many upstream calls, invisible to inbound counting.
• Transport-level sheds can abort agent client loops; internal callers (replay) never traverse the listener.

Dedicated dispatcher goroutine + FIFO queue struct per upstream, requests as first-class objects with deadlines, identity, priority — mirroring mcp-go’s own stdio worker pool (5 workers / queue 100).

Pros

• Richest future capabilities: queue introspection in the Web UI, per-user fair queuing, priorities, cancellation by id.

Cons

• Substantially more code and state for behavior two semaphores already deliver (semaphore waiters are FIFO — fairness is equal today).
• New failure modes: dispatcher leaks, queue-entry lifecycle bugs, shutdown ordering against the managed-client state machine.
• Per-user fairness — the one thing this buys — is a server-edition concern with no current demand, interacting with unimplemented Spec 074. Speculative now.

03Shed semantics

Stdio does not mean limit=1. mcp-go’s own stdio server runs tools/call through a 5-worker pool (queue 100); the client transport legally pipelines by JSON-RPC id; TS/Python SDKs process concurrently. Docs should recommend max_concurrent_requests: 5 for stdio upstreams, with 1 as the floor for fragile servers.

04Implementation plan

Limiter package (pure, TDD-first)

• internal/upstream/limiter/limiter.go + tests. Acquire(ctx, queueTimeout) (release func(), err): admission TryAcquire fail → ErrQueueFull; run Acquire under timeout fail → release admission, ErrQueueTimeout (wrap ctx.Err to distinguish caller-cancel). Release closure bound to this instance so hot-swaps can’t double-release. Zero-config limiter = no-op passthrough.
• Registry: map[serverName]*atomic.Pointer[Limiter] + global pointer; Update() swaps atomically; in-flight releases drain the old instance harmlessly.

Config fields — follow the 4-point checklist exactly

• Global max_concurrent_requests, queue_size, queue_timeout near CallToolTimeout (config.go:173); per-server tri-state pointers copying the HealthCheckInterval pattern (config.go:521-530). Defaults 0 / 0 / 30s. Validation: non-negative; queue_size>0 requires max>0.
• Explicit DetectConfigChanges clause for the three global fields (config_hotreload.go:102-104 model) — per-server fields are already covered by the Servers DeepEqual.
• Env overrides MCPPROXY_MAX_CONCURRENT_REQUESTS / _QUEUE_SIZE / _QUEUE_TIMEOUT (loader.go:638-690); then make swagger (CI verifies) and docs/configuration.md.

Enforcement at the choke point

• Manager owns the Registry; builds per-server limiters in AddServer/ApplyConfig, global in SetGlobalConfig.
• managed/client.go:640 CallTool: resolve effective values (per-server pointer → global default); acquire per-server, then global (a slow upstream’s queue can’t pin global slots); defer release() both; delegate to coreClient. Do not wrap ListTools or Ping. Reconnect-on-use runs before CallTool, so it never holds a slot.

Shed semantics + observability

• Map ErrQueueFull/ErrQueueTimeout via errors.Is in mcp.go (isError tool result, early-error pattern at :2074-2076) and httpapi (429 + Retry-After).
• Activity event status rejected; rejection counter + queue-depth gauge next to existing tool-call metrics (observability/metrics.go:114-129, bridge at observability_bridge.go:16-66). Follow-up: sustained saturation → degraded in the health calculator.

Tests + rollout

• Unit: concurrent acquire/release, instant reject on full queue, queue-timeout, hot-swap under load, zero-config no-op. Hot-reload: extend global_config_hotreload_test.go.
• E2E: slow stdio server with max=1/queue=1 — call 2 queues, call 3 sheds with isError; a code_execution case proving the bypass path is limited. go test -race, -tags server, ./scripts/test-api-e2e.sh.
• Rollout: defaults 0/0 = pure opt-in, ships dark, no migration; hot-reload lets headless operators tune without restarts.

05Open decisions (maintainer input needed)

1. Defaults : keep 0 = unlimited everywhere (recommended, zero behavior change) vs an out-of-the-box stdio default (e.g. 5, mirroring mcp-go’s worker pool) that changes behavior on upgrade.
2. Protocol-level shed : is isError + REST 429 enough, or also emit a -32029 JSON-RPC error with data.retryAfter for non-tool methods (emerging convention; unverified how Claude Code/Cursor react mid-loop)?
3. Literal guarantee : should upstream tools/list and Ping ever count against a strict per-upstream limit for fragile single-threaded stdio servers (sketch excludes them)?
4. Activity vocabulary : new rejected status (touches Web UI filters, possibly telemetry schema) vs reusing error with a distinguishing code.
5. Per-user fairness (server edition): acceptable for v1 that one noisy user can occupy a FIFO queue, or reserve a (user,server) key shape in the Registry API now, ahead of Spec 074?
6. queue_timeout scope : per-server override for symmetry (as sketched) or global-only for a smaller config surface?
7. Health integration : saturation → degraded in v1 or follow-up?

Method: multi-agent workflow (codebase choke-point audit + external prior-art research: Envoy, nginx, LiteLLM, MCP spec 2025-06-18, mcp-go v0.57.0 internals · 1.2M tokens). Companion report: macOS auto-updater for issue #957.