ContextQMD
Back to Semaphore

Performance Research: Many Concurrent Tasks & Large Inventory (5000 hosts)

AGENTS/research/performance-bottlenecks.md

2.19.730.7 KB
Original Source

Performance Research: Many Concurrent Tasks & Large Inventory (5000 hosts)

Date: 2026-06-04 Scope: Go backend of Semaphore (semaphoreui/semaphore). Question: Where are the performance/scalability bottlenecks when (A) many tasks are running concurrently, and (B) an inventory is very large (~5000 hosts)? Method: Static read-through of the task pool, output/streaming path, SQL layer + migrations, API handlers, runner polling protocol, and inventory lifecycle. Every "Critical/High" finding below was re-read against source and the exact file:line confirmed.

TL;DR — the shape of the problem

The system is architected around in-memory task state scanned with O(n) loops, per-second polling loops (both runner→server and the server's own RemoteJob.Run), and a synchronous, per-line, per-user websocket fan-out on the subprocess reader goroutine. None of these matter at 5 tasks / 50 hosts. All of them bite hard at hundreds of tasks / 5000 hosts.

Three structural decisions amplify everything else:

  1. MaxParallelTasks now defaults to 9999 (effectively unlimited — util/config.go:150,390, commit 42e6c00d). The only cheap global admission gate is gone.
  2. The MySQL/Postgres connection pool is never bounded (db/sql/SqlDb.go:82-84 sets a limit only for SQLite). Unlimited parallel tasks × unbounded pool → DB connection exhaustion.
  3. Hot lookups scan or full-copy collections that used to be small: the in-memory running/queued sets, and the unindexed runner.token / task.status / task.created columns.

Top findings, ranked

#FindingSeverityScenarioLocation
1Synchronous per-line, per-user websocket marshal on the subprocess reader goroutineCriticalMany tasks / big outputservices/tasks/TaskRunner_logging.go:27-54
2Runner poll auth = full table scan of unindexed runner.token, every poll, every runnerCriticalMany runnersapi/runners/runners.go:37, db/sql/global_runner.go:12-32
3Per-record db.StoreSession inside the "batched" log writer (connect/close per line if not permanent conn)HighBig outputservices/tasks/TaskPool.go:300-352
4unique(task_id, time) constraint → a batch with 2 same-timestamp lines fails and is silently droppedHighBig outputdb/sql/SqlDb.go:93
5Every queue event re-scans the whole queue O(n²) through one goroutine, calling GetProject (DB) per candidateHighMany tasksservices/tasks/TaskPool.go:206-268,463-496
6RemoteJob.Run busy-polls every 1s per running task; GetTask is an O(n) scan (+ a DB read per task in HA)HighMany remote tasksservices/tasks/RemoteJob.go:209-251, TaskPool.go:142-166
7Full inventory body JSON-marshaled + RSA-encrypted on every runner poll while a job is "starting"HighLarge inventory + runnersapi/runners/runners.go:138-262
8MySQL/Postgres connection pool unbounded; unlimited parallel tasks can exhaust DB connectionsHighMany tasksdb/sql/SqlDb.go:82-84
9GetTaskOutput JSON endpoint loads the entire output table for a task, unpaginatedHighBig outputapi/projects/tasks.go:237, db/sql/task.go:375-397
10Unlimited default parallelism + time.Sleep(1s) per task goroutine + ~3 goroutines/taskHighMany tasksutil/config.go:150,390, TaskPool.go:367-370
11task.status, task.created, task__output.stage_id unindexed; used in filters/sort/retentionHighLarge historydb/sql/task.go:152-160,292, SqlDb.go:934-955
12Per-poll TouchRunner UPDATE + per-request TouchSession UPDATE (write on every poll/request)Medium-HighMany runners / UI clientsapi/runners/runners.go:111-123, api/auth.go:264-268
13getTemplates correlated subquery + N+1 env/vault loads per templateHighMany templatesdb/sql/template.go:283,406-412
14Single RWMutex + full-collection copies (QueueRange/RunningRange) under high-frequency contentionMediumMany tasksservices/tasks/task_state_store.go:121-210
15Remote runner buffers the entire task output in an uncapped in-memory sliceHighBig output (runner)services/runners/running_job.go:46-57, job_pool.go:288-339
16Synchronous, timeout-less alert HTTP/SMTP calls in the status-change hot pathMediumMany tasks finishingservices/tasks/TaskRunner_logging.go:114-131, alert.go
1710 MB bufio.Scanner buffer + 100k-deep channel allocated per pipe per task; over-long line kills the taskMediumMany tasks / big linesservices/tasks/TaskRunner_logging.go:153-201
18GetTaskStats GROUP-BY aggregation over whole task history, no date floor, no cacheMediumLarge historydb/sql/SqlDb.go:934-955
19Stray fmt.Println on every inbound websocket frame (stdout lock contention)LowMany clientsapi/sockets/handler.go:91-92

Scenario A — Many tasks running concurrently

The damage is concentrated in three loops and one fan-out, all of which were sized for "a handful of tasks":

  1. The scheduler is a single goroutine doing O(n²) work. handleQueue (TaskPool.go:206-268) is the only consumer of the unbuffered queueEvents channel. On every event — new task, finished task, requeue, and a 5-second tick — it re-scans the entire waiting queue from index 0. For each candidate it calls blocks() (TaskPool.go:463-496), which issues a GetProject DB query whenever the project has any active task. A burst of N completions = ~N events × O(N) scan × a DB round-trip per candidate. Producers block on the unbuffered channel until this one goroutine drains, so the scheduler becomes a global serialization point. Finding 5.

  2. Each running remote task spins its own 1-second poll. RemoteJob.Run (RemoteJob.go:209-251) does time.Sleep(1s) → GetTask(id) forever. GetTask (TaskPool.go:142-166) full-copies the queue and the running map (QueueRange()/RunningRange(), task_state_store.go:165-210) and linear-scans them. In HA it also does a GetTask DB read per task per second. 300 remote tasks ⇒ ~300 DB reads/sec + ~300×(Q+R) pointer copies/sec of pure bookkeeping. Finding 6.

  3. Runner polls scan the global running set. Each runner's GET poll (runners.go:138) calls GetRunningTasks() ( full map copy under lock) and filters client-side; runner selection (RemoteJob.go:164-179) calls GetNumberOfRunningTasksOfRunner (another full scan) per candidate runner. O(runners × running) per task start. * Findings 3-ish, 14.*

  4. Output fan-out throttles Ansible itself. Every stdout line calls sendToWs (TaskRunner_logging.go:41-54) * before* persistence, marshaling JSON once per user (project users + all admins) and pushing onto the * unbuffered* hub broadcast channel. With many concurrent chatty tasks, the single hub goroutine + per-user marshal is O(lines × users × connections), and backpressure propagates up the channel chain to bufio.Scanner, stalling the subprocess. Findings 1, 14.

Amplifiers: unlimited default parallelism removes the global cap (Finding 10), the unbounded DB pool lets that translate into connection exhaustion (Finding 8), and the per-record StoreSession in the log writer (**Finding 3 **) plus per-poll/per-request writes (Finding 12) pile writes onto the same DB — devastating on SQLite's single writer lock.

Scenario B — Large inventory (5000 hosts)

Good news first (a dedicated agent traced the whole inventory lifecycle): Semaphore treats the inventory as an * opaque blob*. It is never parsed, split per-host, or iterated in Go. There are no per-host loops, no quadratic string building, and no per-host key/file/syscall operations. SSH/become keys are installed once per inventory through a single in-process SSH agent regardless of host count (services/tasks/LocalJob_inventory.go:15-28, pkg/ssh/agent.go:170-212). A static inventory is materialized with a single os.WriteFile ( LocalJob_inventory.go:92-98). Per-host connection fan-out is Ansible's concern (forks), not Semaphore's. So the " 5000 syscalls / O(hosts²)" class of bug does not exist on the Semaphore side.

The real inventory cost is carrying the blob and re-serializing it:

  • Remote runner path (Finding 7) is the only place that genuinely hurts. A 5000-host static inventory is ~0.5–2 MB of text. On every GetRunner poll, the server walks all running tasks and, for any task still in TaskStartingStatus, appends the entire db.Inventory body into RunnerState.NewJobs, then JSON-encodes ( byte-escapes) and chunked-RSA-encrypts the whole payload (runners.go:138-262). A starting task survives several polls until the runner grabs it, so the same multi-MB blob is re-encoded and re-encrypted every second, by every polling runner. Cost ≈ O(inventory_bytes × polls × runners) of alloc + escape + public-key crypto. This is the headline inventory bottleneck.
  • Local static path: one os.WriteFile + one []byte(string) copy per run (LocalJob_inventory.go:92). O(bytes) once per task, not cached between identical runs. Low. (Finding, ranked Low.)
  • Local file inventory: cheapest — Semaphore only joins a path; Ansible reads the file (LocalJob.go:352-357). Touches zero host bytes.
  • Big output is correlated with big inventory: a 5000-host run emits 10⁵+ stdout lines, which is exactly what makes Findings 1, 3, 4, 9, 15, 17 fire. At 5000 hosts, the output path — not the inventory blob — is the dominant cost on the local execution path.

Detailed findings by subsystem

1. Task pool & concurrency (services/tasks/)

  • O(n²) scheduler through one goroutine + DB in the hot loop (Finding 5). handleQueue (TaskPool.go:206-268) re-scans the full queue per event; blocks() (TaskPool.go:463-496) calls store.GetProject per candidate once a project has any active task. Unbuffered queueEvents (one reader) makes producers block on the scheduler. Fix: index "ready" tasks per project/template for O(1) pop; cache project.MaxParallelTasks/ Template.AllowParallelTasks in memory; buffer/coalesce queueEvents; break the inner loop once the parallel cap is hit.
  • 1s busy-poll + O(n) GetTask per remote task (Finding 6). RemoteJob.go:209-251 + TaskPool.go:142-166. QueueRange()/RunningRange() (task_state_store.go:165-210) allocate a full copy each call. Fix: add O(1) GetByID backed by the existing running map[int] (task_state_store.go:124); replace the poll with an event/condition signaled by the runner's progress PUT (which already calls SetStatus).
  • O(running) scan per runner candidate (Finding, Medium). GetNumberOfRunningTasksOfRunner (TaskPool.go:129-136) inside the runner-selection double loop (RemoteJob.go:164-179). Fix: maintain a map[runnerID]int updated in onTaskRun/onTaskStop.
  • Slice-splice dequeue → O(n²) drain (Finding, Medium). DequeueAt (task_state_store.go:154-163) shifts the tail per removal; StopTasksByTemplate (TaskPool.go:583-612) removes one-by-one. Fix: O(1) removal structure (linked list / tombstone-and-compact).
  • Single RWMutex + full-collection copies (Finding 14). MemoryTaskStateStore (task_state_store.go:121-127) guards everything with one lock; RunningRange/QueueRange/Snapshot copy the whole collection while holding it. Not held across I/O (good), but taken at very high frequency by Findings 5/6 and the cluster dashboard Snapshot(). Fix: finer-grained locks or sync.Map; eliminate the high-frequency full-copy callers (Findings 5/6) so the lock is taken far less.
  • Unlimited default parallelism + per-task Sleep (Finding 10). MaxParallelTasks default 9999 ( config.go:150,390); runTask parks a goroutine for time.Sleep(1s) before task.run() (TaskPool.go:367-370); each local task also spawns 2 logPipe goroutines each spawning another (TaskRunner_logging.go:64-65,158) with 100k-buffered channels. Fix: a bounded worker pool / semaphore with a sane default instead of 9999; drop the unconditional time.Sleep.
  • Synchronous, timeout-less alerts in the status hot path (Finding 16). SetStatus → sendMailAlert/ sendTelegramAlert/… (TaskRunner_logging.go:114-131), each a blocking http.Post/SMTP with the default ( no-timeout) client (alert.go:171-487); sendMailAlert re-fetches users from DB per alert (alert.go:81); alertInfos() can panic on a DB error (alert.go:531). A slow endpoint pins the task's goroutine, delaying EventTypeFinished and keeping the task in the running set (inflating every scan above). Fix: bounded background worker for alerts; explicit http.Client timeouts; reuse already-loaded user list.

Verified-OK: log batching design (handleLogs flushes by size 500 / every 500ms, TaskPool.go:271-298); locks are never held across I/O; HA TryClaim/DeleteClaim placement; the schedule pool (robfig/cron, lock held only during Refresh).

2. Task output capture & streaming

  • Synchronous per-line, per-user marshal on the reader goroutine (Finding 1, Critical). sendToWs ( TaskRunner_logging.go:41-54) marshals once per user and runs before the line is queued for DB. t.users = all project users + all admins (TaskRunner.go:377-395). No "is anyone watching this task?" check. Fix: marshal once per line; make delivery async + lossy off the reader goroutine; skip fan-out when no subscriber for that task/project; let the hub filter by user.
  • Unbuffered hub broadcast channel, single goroutine, O(all-connections) per message (Finding, High). api/sockets/pool.go:48-87. The per-connection default: drop is correct (one slow client can't block), but the unbuffered broadcast blocks every caller until the single hub goroutine picks up, and it re-scans all connections even for a user-targeted message. Fix: buffer + lossy broadcast; index connections by userID.
  • Per-record StoreSession inside the batch writer (Finding 3, High). writeLogs (TaskPool.go:300-352) wraps stage_parsers.MoveToNextStage in db.StoreSession per record (lines 314-341). StoreSession ( db/Store.go:790-800) does Connect()/Close() around the callback unless PermanentConnection() — i.e. a DB connect/close per output line on the single shared handleLogs goroutine. In this build MoveToNextStage is a no-op stub but still pays the wrapper cost per line; the Pro build does real per-line DB work. Fix: hoist the session out of the loop (one session per flush); only run stage parsing for stage-using apps; batch its DB effects.
  • unique(task_id, time) drops whole batches (Finding 4, High + correctness). SqlDb.go:93 SetUniqueTogether("task_id","time"); timestamps are per-line tz.Now(). At 5000 hosts many lines share a time; a single InsertTaskOutputBatch (task.go:244-264) is one multi-row INSERT, so a duplicate-timestamp pair fails the * entire 500-line batch*, which is then merely logged and silently dropped (TaskPool.go:347-350). Data loss + perf cliff + log spam. Fix: drop the unique constraint (rows are keyed by autoincrement id); order output by id, not time; fall back to per-row insert on batch failure.
  • Unbounded GetTaskOutput JSON endpoint (Finding 9, High). api/projects/tasks.go:237 calls GetTaskOutputs(..., RetrieveQueryParams{}); with Count==0 no LIMIT is applied (task.go:386-388), so the entire task__output for the task is loaded and marshaled into one JSON array. (The raw endpoint tasks.go:258-298 paginates in 10000-row chunks — the JSON one does not.) Fix: paginate (keyset WHERE task_id=? AND id > ?), or point the UI at the streaming raw endpoint; add composite (task_id, time, id) index.
  • Remote runner buffers entire output in memory (Finding 15, High). running_job.go:46-57 appends every line to an uncapped logRecords; sendProgress (1s ticker, job_pool.go:179,288-296) PUTs the whole slice and only trims after success — a slow/unreachable server ⇒ unbounded growth; the server then replays every record through the synchronous per-line fan-out of Finding 1 in bursts (runners.go:316-318). Fix: cap/ring-buffer logRecords; chunk the payload; feed the receiver into the async coalesced broadcast path.
  • 10 MB scanner buffer + 100k channel per pipe per task; over-long line kills the task (Finding 17, Medium). TaskRunner_logging.go:153-201: make([]byte, 10MB) per pipe (stdout+stderr) per task; scanner.Text() copies a new string per line (GC churn ∝ output); a >10MB line triggers "token too long" and aborts the run (:188-192). Fix: start the buffer small and let it grow; truncate over-long lines instead of killing the task; shrink the 100k channel once the downstream (Findings 1, 3) is fixed.
  • Stray fmt.Println per inbound ws frame (Finding 19, Low). api/sockets/handler.go:91-92 — unconditional stdout write (serialized on the stdout lock) for every inbound frame from every client. Looks like leftover debug. Fix: delete it.

3. Database query patterns & indexes (db/sql/)

Index inventory. EXISTS: task(template_id|project_id|integration_id|inventory_id|schedule_id); task__output(task_id) and (time); event(project_id|user_id); runner(project_id) and (registration_token). * MISSING on hot columns:* runner(token), task(status), task(template_id, created), task(project_id, created), task(template_id, id), task__output(task_id, stage_id), task__output(task_id, time, id). (Note: the task__output(task_id) index the brief worried about does exist — migrations/v2.15.1.sqlite.sql:413, v2.17.15.sql:6. Good.)

  • runner.token full scan on every poll (Finding 2, Critical). RunnerMiddleware (runners.go:37) → GetRunnerByToken (global_runner.go:12-32) → SELECT * FROM runner WHERE token=? (SqlDb.go:491-501); no project_id predicate (global), no index on token (only project_id and registration_token exist). Bolt is O(N) in memory (bolt/global_runner.go:12-30). With N runners polling ~1/s and a PUT+GET per cycle, that is ~2N full scans/sec. Fix: unique index on runner(token); in-memory token→runner cache with short TTL.
  • task.status / task.created unindexed (Finding 11, High). Status filter (task.go:292), GetTaskStats group/filter on created,status,start,end,user_id (SqlDb.go:934-955), and clearTasks ORDER BY created on the task-creation write path (task.go:152-160). Fix: add task(status) (or task(project_id,status)), task(template_id,created), task(project_id,created).
  • getTemplates correlated subquery + N+1 (Finding 13, High). Per-template (SELECT id FROM task WHERE template_id=pt.id ORDER BY id DESC LIMIT 1) (template.go:283) + a GetTemplateEnvironments and GetTemplateVaults query per template (template.go:406-412, loadVaults=true). 200 templates ⇒ 400–600 queries/page. Fix: batch env/vault with WHERE template_id IN (...); add task(template_id, id) for an index-only backward scan of last_task_id.
  • getTasks N+1 Fill() (Finding, Medium/High for deploy projects). task.go:314-319 loops Fill(); TaskWithTpl.Fill (db/Task.go:184-196) does a GetTask per row with a non-nil BuildTaskID. Up to ~1000 extra joins per history page. Fix: resolve all BuildTaskIDs in one WHERE id IN (...).
  • clearTasks retention on the write path (Finding, Medium). Runs synchronously inside CreateTask ( task.go:121-168): a rand-gated count(*), an ORDER BY created scan, and a range DELETE — all without task(created). Fix: the (template_id, created) index; move retention to a background sweep; delete by id cutoff (monotonic, index-backed).
  • FillEvents N+1 (Finding 7-DB, Medium). db/Event.go:73-114 calls GetTask per task-event (usernames are memoized, tasks are not). Hundreds of joins per activity feed. Fix: batch-resolve task object names with IN; memoize like usernames.
  • GetTaskStats unbounded aggregation, no cache (Finding 18, Medium). SqlDb.go:934-955 groups the whole project history with no date floor when start is absent. Fix: short-TTL cache per (project, template, range); default a bounded window; covering index (project_id, template_id, created, status).
  • MySQL/Postgres pool unbounded (Finding 8, High). SqlDb.go:82-84 sets SetMaxOpenConns(1) for SQLite only; nothing for MySQL/Postgres. Go default = unlimited open / 2 idle. Unlimited parallel tasks each writing status/output/stage/event ⇒ connection-count blowup + constant idle churn. Fix: set SetMaxOpenConns/SetMaxIdleConns/SetConnMaxLifetime (configurable) right where the SQLite branch is.

4. API layer & runner polling

  • runner.token scan + GetRunningTasks per poll (Findings 2, 3-API, 7). Covered above; the GET poll also ranges the global running set (runners.go:138, GetRunningTasks full-copies the map) and, for "starting" tasks, decrypts every secret + JSON + RSA-encrypts the whole payload including the inventory body, every second per runner even when there is no new job. Fix: index the running set by runner id; mark a job "dispatched" so it's built/encrypted once; long-poll or push instead of 1s poll; ship inventory by id+hash, not inline.
  • Per-poll / per-request writes (Finding 12, Medium-High). TouchRunner UPDATE on every GET poll ( runners.go:111-123, only needed at 30-min granularity per RemoteJob.go:23); TouchSession UPDATE on every authenticated request (auth.go:264-268, last_active only used to expire >7-day sessions). N runners + many UI tabs ⇒ steady write stream; brutal on SQLite's single writer. Fix: debounce both via an in-memory last-touch map (write only if older than ~60s / ~5min).
  • Admin "all tasks" + per-project task list, no pagination/cache (Findings 5-API, 6-API, Medium). api/tasks/tasks.go:43-71 copies the whole queued+running set per poll (admin Tasks view polls every 10s, Tasks.vue:92); GetAllTasks fetches up to 1000 joined rows with a conditional N+1 (tasks.go:100, task.go:269-322). Fix: pagination (keyset on id desc); serve admin list from a rate-limited cached snapshot or push over websocket.
  • No read-path cache anywhere (Finding 9-API, Low-but-enabling). api/cache.go only clears the tmp dir ( synchronously, in-handler). Every hot read hits the DB. Fix: small in-process TTL cache for token→runner, userID→user, session validity; run ClearTmpDir in a goroutine.
  • getAllEvents unbounded (Finding 10-API, Low). api/events.go:30-49 passes Count:0 ⇒ no LIMIT ⇒ entire events table for the project/user. Fix: default a sane limit + pagination; index event(project_id, id desc).

Verified-OK: SQL PermanentConnection() is true so StoreMiddleware is a no-op per request (SqlDb.go:563); runner-tag loading is a single IN batch (runner_tag.go:13-52); output write batching is efficient.

5. Large inventory — confirmation of what is not a problem

Static inventory: one os.WriteFile per run (LocalJob_inventory.go:92-98). SSH/become keys: one per inventory via one in-process agent (LocalJob_inventory.go:15-28, pkg/ssh/agent.go:170-212). Vars/secrets: built once as process env + one --extra-vars JSON, independent of host count (LocalJob.go:451-464). No per-host loop, no quadratic string building, no per-host syscall anywhere in Go. The only inventory-scaling cost is the blob being carried by value through 4–5 hops (cheap until marshaled) and the runner-poll re-serialization of Finding 7.

Cross-cutting root causes

  1. Polling instead of events. Both RemoteJob.Run (server-side, per task) and the runner job pool (client-side) poll every 1s; the UI polls every 10s. Each poll re-does full work (scans, full payload build, heartbeat write). The runner already PUTs progress — status transitions and job hand-off should be event/long-poll driven, not re-derived every second.
  2. O(n) scans/copies of in-memory state for the most frequent operations. Status lookup, scheduling, runner load, and the admin list all full-scan or full-copy the queue/running collections under one lock. These need an id-keyed map and per-runner counters.
  3. The cheap global cap was removed without adding a real one. MaxParallelTasks=9999 + unbounded DB pool means " many tasks" now translates directly into goroutine count, DB connections, and contention with no backstop.
  4. Missing indexes on the columns the hot queries actually use (runner.token, task.status, task.created, task__output.stage_id).
  5. The output path does per-line work that should be per-batch (per-user marshal, per-record StoreSession) and persists with a constraint (unique(task_id,time)) that high-volume output structurally violates.

Prioritized remediation roadmap

Phase 0 — Cheap, high-leverage (migrations + a few lines)

  1. Add indexes: runner(token) (unique), task(status) (or task(project_id,status)), task(template_id,created), task(project_id,created), task(template_id,id), task__output(task_id,stage_id), task__output(task_id,time,id). (Findings 2, 11, 13, 18, 9)
  2. Bound the MySQL/Postgres pool at db/sql/SqlDb.go:82 (SetMaxOpenConns/Idle/Lifetime, configurable). (Finding 8)
  3. Drop the unique(task_id, time) constraint, order output by id, and fall back to per-row insert on batch failure. (Finding 4)
  4. Hoist db.StoreSession out of the writeLogs loop (one session per flush). (Finding 3)
  5. Debounce TouchRunner and TouchSession via in-memory last-touch maps. (Finding 12)
  6. Delete the fmt.Println in the websocket handler. (Finding 19)

Phase 1 — Output & streaming (the 5000-host local path)

  1. Marshal each log line once, deliver async + lossy off the reader goroutine, skip fan-out when no subscriber; index hub connections by user. (Findings 1, 14-hub)
  2. Paginate GetTaskOutput (keyset) or route the UI to the streaming raw endpoint. (Finding 9)
  3. Cap the runner's in-memory logRecords and chunk sendProgress. (Finding 15)
  4. Grow the scanner buffer lazily; truncate over-long lines instead of aborting. (Finding 17)

Phase 2 — Scheduling & polling (the many-tasks path)

  1. O(1) GetByID in the state store (back the running set with the existing id map); add a map[runnerID]int load counter. (Findings 6, runner-select)
  2. Replace the 1s RemoteJob.Run poll with an event/condition signaled by the runner progress PUT. (Finding 6)
  3. Cache project/template parallel-task limits; index "ready" tasks so handleQueue pops in O(1); buffer/coalesce queueEvents. (Finding 5)
  4. Reconsider unlimited default parallelism — use a bounded worker pool with a sane default; drop the per-task time.Sleep(1s). (Finding 10)
  5. Long-poll / push the runner job hand-off; build+encrypt JobData once per job; ship inventory by id+hash, not inline per poll. (Finding 7)

Phase 3 — DB hygiene & dashboards

  1. Batch the N+1s (getTemplates env/vault, getTasks.Fill, FillEvents) with IN (...). (Findings 13, getTasks, FillEvents)
  2. Move task retention off the write path (clearTasks), delete by id cutoff. (clearTasks)
  3. Cache dashboard stats (short TTL) and default a bounded date window. (Finding 18)
  4. Fan alerts out to a bounded background worker with HTTP timeouts. (Finding 16)

Suggested measurements (to confirm before/after)

  • pprof CPU + goroutine profile of the server while running ~200 concurrent tasks emitting heavy output; expect sendToWs/json.Marshal, handleQueue/GetProject, and RemoteJob.Run/GetTask to dominate.
  • DB: enable slow-query log; expect SELECT * FROM runner WHERE token=?, the GetTaskStats GROUP BY, and clearTasks ORDER BY created to appear. Watch active connection count vs. max_connections.
  • A single 5000-host task on a remote runner: capture the GetRunner response size and per-poll CPU; expect multi-MB JSON + RSA cost repeated per poll until pickup.
  • Memory: heap profile during many concurrent tasks; expect the 10 MB scanner buffers and 100k channels (Finding 17) and uncapped runner logRecords (Finding 15) to show.

Appendix — verification status

Re-read and confirmed against source by the primary researcher (not just sub-agent report): Findings 1 ( TaskRunner_logging.go:27-54), 2 (runner.token indexes across all migrations + runners.go:37), 3 ( TaskPool.go:300-352), 4 (SqlDb.go:93), 7 (runners.go:138-262), 8 (SqlDb.go:82-84), 10 (config.go:150,390 + TaskPool.go:367-370). The remaining findings come from focused sub-agent read-throughs with file:line citations and were cross-checked where they overlapped (the unbounded DB pool, the 1s poll loops, and the inventory-in-poll serialization were each independently reported by multiple agents). Line numbers reflect the working tree on branch develop as of 2026-06-04; a few may drift by a line or two after edits.