docs/ref/modules/utils/sync-protocol/lifecycle.md
The Agent Sync Protocol implements a session-based synchronization mechanism with well-defined phases and message types. This document details the complete lifecycle of a synchronization session, including all message types and state transitions.
The protocol operates through three main phases:
Initiates a synchronization session.
Direction: Agent → Manager
Content:
FlatBuffer Schema:
table Start {
mode: Mode;
size: uint64;
}
State Transition: Idle → WaitingStartAck
Acknowledges session establishment and provides session ID.
Direction: Manager → Agent
Content:
FlatBuffer Schema:
table StartAck {
status: Status;
session: uint64;
}
State Transition: WaitingStartAck → DataTransfer
Transmits individual differences to the manager.
Direction: Agent → Manager
Content:
FlatBuffer Schema:
table DataValue {
seq: ulong;
session: ulong;
operation: Operation;
id: string;
index: string;
version: ulong;
data: [byte];
}
State: Remains in DataTransfer
Notifies the manager that specific indices should be cleaned.
Direction: Agent → Manager
Content:
FlatBuffer Schema:
table DataClean {
seq: ulong;
session: ulong;
index: string;
}
State: Remains in DataTransfer
Usage: Sent during data cleaning synchronization mode when notifying the manager that specific indices have been disabled or should be cleared.
Signals completion of data transmission.
Direction: Agent → Manager
Content:
FlatBuffer Schema:
table End {
session: ulong;
}
State Transition: DataTransfer → WaitingEndAck
Manager requests retransmission of specific data ranges.
Direction: Manager → Agent
Content:
FlatBuffer Schema:
table ReqRet {
seq: [Pair];
session: ulong;
}
table Pair {
begin: ulong;
end: ulong;
}
State: Remains in DataTransfer
Agent Response: Retransmits requested DataValue messages
Acknowledges an End message. May be sent multiple times for the same session with different statuses.
Direction: Manager → Agent
Content:
FlatBuffer Schema:
table EndAck {
status: Status;
session: ulong;
}
Status Values:
Ok: Session completed successfully; agent deletes the synced data from the persistent queue.Error: Session failed (e.g. checksum mismatch); agent does not delete data.Processing: Manager received the End message but the session is still being processed (e.g. queued for indexing). The agent must wait again without resending End and without consuming a retry attempt.State Transitions:
WaitingEndAck → Idle on Ok or ErrorWaitingEndAck → WaitingEndAck on Processing (reset wait, no retry consumed)Transmits checksum information for integrity verification.
Direction: Agent → Manager
Content:
FlatBuffer Schema:
table ChecksumModule {
session: ulong;
index: string;
checksum: string;
}
State: Used during integrity check mode, sent between StartAck and End
The requiresFullSync method implements a specialized synchronization flow for checksum verification:
Agent Manager
| |
|-------------- Start ----------------> |
| (mode=CHECK, checksum) |
| |
|<------------ StartAck ---------------- |
| (session_id) |
| |
|---------- ChecksumModule -----------> |
| (index, checksum) |
| |
|--------------- End ------------------> |
| (session_id) |
| |
|<------------- EndAck ----------------- |
| (status: match/mismatch) |
| |
Process:
Error if mismatch (full sync needed), Ok if validThe synchronizeMetadataOrGroups method implements a simplified flow without data transfer:
Agent Manager
| |
|-------------- Start ----------------> |
| (mode=METADATA_DELTA/GROUP_DELTA) |
| |
|<------------ StartAck ---------------- |
| (session_id) |
| |
|--------------- End ------------------> |
| (session_id) |
| |
|<------------- EndAck ----------------- |
| (success) |
| |
Supported Modes:
METADATA_DELTA: Metadata delta synchronizationMETADATA_CHECK: Metadata integrity checkGROUP_DELTA: Group delta synchronizationGROUP_CHECK: Group integrity checkProcess:
The notifyDataClean method implements a specialized flow for notifying the manager about data cleaning when modules are disabled:
Agent Manager
| |
|-------------- Start ----------------> |
| (mode=DELTA, size=N, indices=[...]) |
| |
|<------------ StartAck ---------------- |
| (session_id) |
| |
|---------- DataClean[0] --------------> |
| (seq=0, session, index="fim_files") |
| |
|---------- DataClean[1] --------------> |
| (seq=1, session, index="fim_registry")|
| |
| ... |
| |
|---------- DataClean[N-1] ------------> |
| (seq=N-1, session, index=...) |
| |
|--------------- End ------------------> |
| (session_id) |
| |
|<------------- EndAck ----------------- |
| (status: Ok) |
| |
| clearItemsByIndex() for each index |
| (cleanup local database entries) |
| |
When using persistDifferenceInMemory for recovery scenarios:
Agent (Recovery) Manager
| |
| clearInMemoryData() |
| (cleanup before sync) |
| |
| persistDifferenceInMemory() × N |
| (storing recovery data in memory) |
| |
|-------------- Start ----------------> |
| (mode=FULL) |
| |
|<------------ StartAck ---------------- |
| |
|------- DataValue (from memory) ------> |
|------- DataValue (from memory) ------> |
| ... |
| |
|--------------- End ------------------> |
| |
|<------------- EndAck ----------------- |
| |
Process:
clearInMemoryData() to make sure memory is clean before persistingpersistDifferenceInMemory()Agent Manager
| |
|-------------- Start ----------------> |
| (mode, count) |
| |
|<------------ StartAck ---------------- |
| (session_id) |
| |
|----------- DataValue[0] -------------> |
|----------- DataValue[1] -------------> |
|----------- DataValue[2] -------------> |
| ... |
|----------- DataValue[N] -------------> |
| |
|--------------- End ------------------> |
| (session_id) |
| |
|<------------- EndAck ----------------- |
| (success) |
| |
When the manager receives End and immediately enqueues the session for indexing, it sends EndAck(Processing) to keep the agent waiting. The agent must not resend End and must not count this as a failed attempt.
Agent Manager
| |
|-------------- Start ----------------> |
| |
|<------------ StartAck ---------------- |
| |
|----------- DataValue[0..N] ----------> |
| |
|--------------- End ------------------> |
| |
|<------- EndAck(Processing) ----------- | (session queued, not yet indexed)
| (wait again, no retry consumed, |
| End NOT resent) |
| |
|<------------- EndAck(Ok) ------------- | (indexing complete)
| |
Agent Manager
| |
|-------------- Start ----------------> |
| |
|<------------ StartAck ---------------- |
| |
|----------- DataValue[0] -------------> |
|----------- DataValue[1] -------------> |
|----------- DataValue[2] -----X (lost) |
|----------- DataValue[3] -------------> |
|----------- DataValue[4] -------------> |
| |
|--------------- End ------------------> |
| |
|<------------- ReqRet ----------------- |
| (ranges: [[2,2]]) |
| |
|----------- DataValue[2] -------------> | (retransmission)
| |
|<------------- EndAck ----------------- |
| |
stateDiagram-v2
[*] --> Idle
Idle --> WaitingStartAck: Send Start
WaitingStartAck --> DataTransfer: Receive StartAck
WaitingStartAck --> Idle: Timeout/Error
DataTransfer --> DataTransfer: Send DataValue/DataClean
DataTransfer --> WaitingEndAck: Send End
WaitingEndAck --> Idle: Receive EndAck(Ok/Error)
WaitingEndAck --> WaitingEndAck: Receive EndAck(Processing)
WaitingEndAck --> DataTransfer: Receive ReqRet
WaitingEndAck --> Idle: Timeout/Error
Each phase has specific timeout behaviors:
WaitingStartAck
DataTransfer
WaitingEndAck
EndAck(Ok/Error): Session completes or fails immediatelyEndAck(Processing): Manager is still processing; agent resets the wait timer and continues waiting without resending End and without consuming a retryReqRet: Agent retransmits missing sequences; retrying does not consume a retryEnd: Retry sending End message, consuming one retryInvalid Session ID
Unexpected Message Type
Malformed Messages