Worker Commands

Worker commands are server-initiated instructions sent to workers via Nexus. Each worker process has a dedicated control task queue that it polls via Nexus RPC. A single control queue serves one or more workers running within the same process. This enables the server to push actions to workers without relying on heartbeats or long-poll cycles.

To route a command, the server needs to know the target worker's control queue. For commands that target activities, this information is stored directly in the mutable state (ActivityInfo) when the activity was dispatched to the worker.

graph LR
    subgraph History
        A[Create Task] -->|1| C[Outbound Queue]
        C -->|2| D[Dispatch]
    end
    subgraph Matching
        E[Control Queue]
    end
    D -->|3| E
    E -->|7| D
    E ~~~ Worker
    subgraph Worker
        G[Poller] -->|5| F[Execute]
    end
    G -.->|4| E
    F -.->|6| E

History

1. Create Task -- The server decides whether to initiate a worker command based on a state change (e.g., a user requesting activity cancellation). The trigger logic creates one or more WorkerCommand protos and calls GenerateWorkerCommandsTasks, which persists them as an outbound task. Serialization is command-agnostic (task_serializers.go).
2. Outbound Queue -- The outbound queue executor picks up the task and invokes the dispatcher configured in Execute.
3. Dispatch -- The dispatcher sends a DispatchNexusTask RPC to matching. This is synchronous — matching holds the request until a worker responds or the request times out.

Matching

4. Control Queue -- Matching holds the command on the control queue until a worker polls it. If no worker is polling, the request times out. There is currently no way to distinguish a permanently gone worker from a temporarily slow one, so the dispatcher simply retries up to a maximum number of attempts before dropping the task. This is acceptable because worker commands are best-effort.

Worker

5. Poller -- The worker's poller picks up the command from the control queue.
6. Execute -- The worker executes the command and sends the response back to matching.

Matching

7. Response -- Matching forwards the response back to the dispatcher in history.

Adding a new command

1. Define the proto -- Add a new variant to WorkerCommand in message.proto (api repo).
2. Add a trigger -- Add server-side logic to detect the condition, create the command, and call GenerateWorkerCommandsTasks (see task_generator.go).
3. Handle in the SDK -- Add a handler for the new command variant in the SDK's worker command executor.