Admin Worker Plugin System (Design)

This document describes the plugin system for admin-managed workers, implemented in parallel with the current maintenance/worker mechanism.

Scope

• Add a new plugin protocol and runtime model for multi-language workers.
• Keep all current admin + worker code paths untouched.
• Use gRPC for all admin-worker communication.
• Let workers describe job configuration UI declaratively via protobuf.
• Persist all job type configuration under admin server data directory.
• Support detector workers and executor workers per job type.
• Add end-to-end workflow observability (activities, active jobs, progress).

New Contract

• Proto file: weed/pb/plugin.proto
• gRPC service: PluginControlService.WorkerStream
• Connection model: worker-initiated long-lived bidirectional stream.

Why this model:

• Works for workers in any language with gRPC support.
• Avoids admin dialing constraints in NAT/private networks.
• Allows command/response, progress streaming, and heartbeat over one channel.

Core Runtime Components (Admin Side)

1. PluginRegistry

• Tracks connected workers and their per-job-type capabilities.
• Maintains liveness via heartbeat timeout.

2. SchemaCoordinator

• For each job type, asks one capable worker for JobTypeDescriptor.
• Caches descriptor version and refresh timestamp.

3. ConfigStore

• Persists descriptor + saved config values in dataDir.
• Stores both:
  • Admin-owned runtime config (detection interval, dispatch concurrency, retry).
  • Worker-owned config values (plugin-specific detection/execution knobs).

4. DetectorScheduler

• Per job type, chooses one detector worker (can_detect=true).
• Sends RunDetectionRequest with saved configs + cluster context.
• Accepts DetectionProposals, dedupes by dedupe_key, inserts jobs.

5. JobDispatcher

• Chooses executor worker (can_execute=true) for each pending job.
• Sends ExecuteJobRequest.
• Consumes JobProgressUpdate and JobCompleted.

6. WorkflowMonitor

• Builds live counters and timeline from events:
  • activities per job type,
  • active jobs,
  • per-job progress/state,
  • worker health/load.

Worker Responsibilities

1. Register capabilities on connect (WorkerHello).
2. Expose job type descriptor (ConfigSchemaResponse) including UI schemas:

• admin config form,
• worker config form,
• defaults.

3. Run detection on demand (RunDetectionRequest) and return proposals.
4. Execute assigned jobs (ExecuteJobRequest) and stream progress.
5. Heartbeat regularly with slot usage and running work.
6. Handle cancellation requests (CancelRequest) for in-flight detection/execution.

Declarative UI Model

UI is fully derived from protobuf schema:

• ConfigForm
• ConfigSection
• ConfigField
• ConfigOption
• ValidationRule
• ConfigValue (typed scalar/list/map/object value container)

Result:

• Admin can render forms without hardcoded task structs.
• New job types can ship UI schema from worker binary alone.
• Worker language is irrelevant as long as it can emit protobuf messages.

Detection and Dispatch Flow

1. Worker connects and registers capabilities.
2. Admin requests descriptor per job type.
3. Admin persists descriptor and editable config values.
4. On detection interval (admin-owned setting):

• Admin chooses one detector worker for that job type.
• Sends RunDetectionRequest with:
  • AdminRuntimeConfig,
  • admin_config_values,
  • worker_config_values,
  • ClusterContext (master/filer/volume grpc locations, metadata).

5. Detector emits DetectionProposals and DetectionComplete.
6. Admin dedupes and enqueues jobs.
7. Dispatcher assigns jobs to any eligible executor worker.
8. Executor emits JobProgressUpdate and JobCompleted.
9. Monitor updates workflow UI in near-real-time.

Persistence Layout (Admin Data Dir)

Current layout under <admin-data-dir>/plugin/:

• job_types/<job_type>/descriptor.pb
• job_types/<job_type>/descriptor.json
• job_types/<job_type>/config.pb
• job_types/<job_type>/config.json
• job_types/<job_type>/runs.json
• jobs/tracked_jobs.json
• activities/activities.json

config.pb should use PersistedJobTypeConfig from plugin.proto.

Admin UI

• Route: /plugin
• Includes:
  • runtime status,
  • workers/capabilities,
  • declarative descriptor-driven config forms,
  • run history (last 10 success + last 10 errors),
  • tracked jobs and activity stream,
  • manual actions for schema refresh, detection, and detect+execute workflow.

Scheduling Policy (Initial)

Detector selection per job type:

• only workers with can_detect=true.
• prefer healthy worker with highest free detection slots.
• lease ends when heartbeat timeout or stream drop.

Execution dispatch:

• only workers with can_execute=true.
• select by available execution slots and least active jobs.
• retry on failure using admin runtime retry config.

Safety and Reliability

• Idempotency: dedupe proposals by (job_type, dedupe_key).
• Backpressure: enforce max jobs per detection run.
• Timeouts: detection and execution timeout from admin runtime config.
• Replay-safe persistence: write job state changes before emitting UI events.
• Heartbeat-based failover for detector/executor reassignment.

Backward Compatibility

• Legacy worker.proto runtime remains internally available where still referenced.
• External CLI worker path is moved to plugin runtime behavior.
• Runtime is enabled by default on admin worker gRPC server.

Incremental Rollout Plan

Phase 1

• Introduce protocol and storage models only.

Phase 2

• Build admin registry/scheduler/dispatcher behind feature flag.

Phase 3

• Add dedicated plugin UI pages and metrics.

Phase 4

• Port one existing job type (e.g. vacuum) as external worker plugin.

Phase 4 status (starter)

• Added weed worker command as an external plugin.proto worker process.
• Initial handler implements vacuum job type with:
  • declarative descriptor/config form response (ConfigSchemaResponse),
  • detection via master topology scan (RunDetectionRequest),
  • execution via existing vacuum task logic (ExecuteJobRequest),
  • heartbeat/load reporting for monitor UI.
• Legacy maintenance-worker-specific CLI path is removed.

Run example:

• Start admin: weed admin -master=localhost:9333
• Start worker: weed worker -admin=localhost:23646
• Optional explicit job type: weed worker -admin=localhost:23646 -jobType=vacuum
• Optional stable worker ID persistence: weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin

Phase 5

• Migrate remaining job types and deprecate old mechanism.

Agreed Defaults

1. Detector multiplicity

• Exactly one detector worker per job type at a time. Admin selects one worker and runs detection there.

2. Secret handling

• No encryption at rest required for plugin config in this phase.

3. Schema compatibility

• No migration policy required yet; this is a new system.

4. Execution ownership

• Same worker is allowed to do both detection and execution.

5. Retention

• Keep last 10 successful runs and last 10 error runs per job type.