ContextQMD
Back to Temporal

Workflow lifecycle

docs/architecture/workflow-lifecycle.md

1.32.0-155.113.8 KB
Original Source

Workflow lifecycle

Below we follow a typical sequence of events in the execution of the following very simple workflow pseudocode:

javascript
function myWorkflow() {
  result = callActivity(myActivity);
  return result;
}

Note that below, "Initialize history", "Append event to history", or "Persist Mutable State and history tasks", all refer to durable writes to the persistence layer.

  1. The User Application sends a StartWorkflowExecution request
    • Workflow History is initialized with [WorkflowExecutionStarted, WorkflowTaskScheduled]
    • A Workflow Task is added in the Matching service
mermaid
sequenceDiagram
User Application->>Frontend: StartWorkflowExecution
Frontend->> History: StartWorkflowExecution
History ->> Persistence: CreateWorkflowExecution
note over Persistence: Initialize History:
[WorkflowExecutionStarted, WorkflowTaskScheduled]
persist MutableState and Transfer Task
Persistence ->> History: Create Succeed
History->>Frontend: Start Succeed
Frontend->>User Application: Start Succeed
loop QueueProcessor
    History->>Persistence: GetHistoryTasks
		History->>History: ProcessTask
		History->>Matching: AddWorkflowTask

end
mermaid
---
config:
  layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Scheduled"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("WorkflowTask")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Workflow Timeout")
      end

      ms ~~~ transferQ ~~~ timerQ
   end

   subgraph history[Workflow History]
      direction TB
      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled]
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>
  1. A Worker dequeues and processes the Workflow Task
    • It advances the workflow execution, and becomes blocked on the Activity call.
mermaid
sequenceDiagram
Worker->>Frontend: PollWorkflowTask
Frontend->>Matching: PollWorkflowTask
History->>Matching: AddWorkflowTask (step 1 above)
Matching->>History: RecordWorkflowTaskStarted
History->>Persistence: UpdateWorkflowExecution
note over Persistence: append History Event: WorkflowTaskStarted
update Mutable State & add Timer Task (workflow task timeout)
Persistence->>History: Update Succeed
History->>Persistence: GetHistoryEvents
Persistence->>History: History Events
History->>Matching: Record Succeed
Matching->>Frontend: WorkflowTask
Frontend->>Worker: WorkflowTask
Worker->>Worker: Advance workflow
mermaid
---
config:
  layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Started"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("&nbsp")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Workflow Task Timeout") ~~~
         timer2("Workflow Timeout")
      end

      ms ~~~ transferQ ~~~ timerQ
   end

   subgraph history[Workflow History]
      direction TB

      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled] ~~~
      e3[WorkflowTaskStarted]
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>
  1. The workflow launches an activity, causing the Worker to send a ScheduleActivityTask command back to the Frontend
    • An Activity task is added in the Matching service.
mermaid
sequenceDiagram
Worker ->> Frontend: RespondWorkflowTaskCompleted
[ScheduleActivityTask]
Frontend->> History: RespondWorkflowTaskCompleted
[ScheduleActivityTask]
History ->> Persistence: UpdateWorkflowExecution
note over Persistence: append History Events: WorkflowTaskCompleted, ActivityTaskScheduled
update MutableState and add Transfer Task
Persistence ->> History: Update Succeed
History->>Frontend: Respond Succeed
Frontend->>Worker: Respond Succeed
loop QueueProcessor
    History->>Persistence: GetHistoryTasks
		History->>History: ProcessTask
		History->>Matching: AddActivityTask
end
mermaid
---
config:
  layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Empty"] ~~~
         ms2["Activity Task: Scheduled"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("Activity Task")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Workflow Task Timeout") ~~~
         timer2("Activity Task Timeout") ~~~
         timer3("Workflow Timeout")
      end

      ms ~~~ transferQ  ~~~ timerQ
   end

   subgraph history[Workflow History]
      direction TB

      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled] ~~~
      e3[WorkflowTaskStarted] ~~~
      e4[WorkflowTaskCompleted] ~~~
      e5[ActivityTaskScheduled]
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>

4. A Worker dequeues the Activity task and executes the activity

mermaid
sequenceDiagram
Worker->>Frontend: PollActivityTask
Frontend->>Matching: PollActivityTask
History->>Matching: AddActivityTask (step 3 above)
Matching->>History: RecordActivityStarted
History->>Persistence: UpdateWorkflowExecution
note over Persistence: append History Event: ActivityTaskStarted
update Mutable State, add Timer Task (activity task timeout)
Persistence->>History: Update succeed
History->>Matching: Record Succeed
Matching->>Frontend: ActivityTask
Frontend->>Worker: ActivityTask
Worker->>Worker: Execute activity
mermaid
---
config:
  layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Empty"] ~~~
         ms2["Activity Task: Started"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("&nbsp")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Activity Task Timeout") ~~~
         timer2("Workflow Timeout")
      end

      ms ~~~ transferQ ~~~ timerQ
   end

   subgraph history[Workflow History]
      classDef dottedBoxStyle stroke-dasharray: 5 5;
      direction TB

      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled] ~~~
      e3[WorkflowTaskStarted] ~~~
      e4[WorkflowTaskCompleted] ~~~
      e5[ActivityTaskScheduled] ~~~
      e6[ActivityTaskStarted]:::dottedBoxStyle
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>
  1. Once the activity is completed, the Worker running the Activity sends RespondActivityTaskCompleted, containing the activity result
    • A Workflow Task is added in the Matching service
mermaid
sequenceDiagram
Worker->>Frontend: RespondActivityTaskCompleted
Frontend->>History: RespondActivityTaskCompleted
History->>Persistence: UpdateWorkflowExecution
note over Persistence: append History Events:
ActivityTaskCompleted  (w/ activity result), WorkflowTaskScheduled
update Mutable State & add Transfer Task
Persistence->>History: Update Succeed
History->>Frontend: Respond Succeed
Frontend->>Worker: Respond Succeed
loop QueueProcessor
    History->>Persistence: GetHistoryTasks
		History->>History: ProcessTask
		History->>Matching: AddWorkflowTask
end
mermaid
---
config:
  layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Scheduled"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("Workflow Task")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Workflow Task Timeout") ~~~
         timer2("Activity Task Timeout") ~~~
         timer3("Workflow Timeout")
      end

      ms ~~~ transferQ ~~~ timerQ
   end

   subgraph history[Workflow History]
      direction TB

      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled] ~~~
      e3[WorkflowTaskStarted] ~~~
      e4[WorkflowTaskCompleted] ~~~
      e5[ActivityTaskScheduled] ~~~
      e6[ActivityTaskStarted] ~~~
      e7[ActivityTaskCompleted] ~~~
      e8[WorkflowTaskScheduled]
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>
  1. The Worker dequeues the Workflow Task
    • It advances the workflow, and finds that it has reached its end.

<Same sequence diagram as step 2 above>

  1. The Worker sends RespondWorkflowTaskCompleted, containing a CompleteWorkflowExecution command
mermaid
sequenceDiagram
Worker->>Frontend: RespondWorkflowTaskCompleted
[CompleteWorkflowExecution]
Frontend->>History: RespondWorkflowTaskCompleted
[CompleteWorkflowExecution]
History->>Persistence: UpdateWorkflowExecution
note over Persistence: append HistoryEvents: WorkflowTaskCompleted, WorkflowExecutionCompleted
update Mutable State & add tasks (visibility, tiered storage, retention etc)
Persistence->>History: Update Succeed
History->>Frontend: Respond Succeed
Frontend->>Worker: Respond Succeed
loop QueueProcessor
    History->>Persistence: GetHistoryTasks
		History->>History: ProcessTask (Update visibility, Upload to S3, Delete data etc)
end
mermaid
---
config:
  layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Empty"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("&nbsp")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Workflow Task Timeout") ~~~
         timer2("Activity Task Timeout") ~~~
         timer3("Workflow Timeout")
      end

      ms ~~~ transferQ ~~~ timerQ
   end

   subgraph history[Workflow History]
      direction TB

      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled] ~~~
      e3[WorkflowTaskStarted] ~~~
      e4[WorkflowTaskCompleted] ~~~
      e5[ActivityTaskScheduled] ~~~
      e6[ActivityTaskStarted] ~~~
      e7[ActivityTaskCompleted] ~~~
      e8[WorkflowTaskScheduled] ~~~
      e9[WorkflowTaskStarted] ~~~
      e10[WorkflowTaskCompleted] ~~~
      e11[WorkflowExecutionCompleted]
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>

Alternatively, the Activity may fail and be retried:

mermaid
sequenceDiagram
Worker->>Frontend: RespondActivityFailed
Frontend->>History: RespondActivityFailed
History->>Persistence: UpdateWorkflowExecution
note over Persistence: Append History Events: ActivityTaskFailed, ActivityTaskScheduled
update MutableState & add Timer Task (activity timout)
Persistence->>History: Update Succeed
History->>Frontend: Respond Succeed
Frontend->>Worker: Respond Succeed
loop QueueProcessor
    History->>Persistence: GetHistoryTasks
		History->>History: ProcessTask
		History->>Matching: AddActivityTask
end
mermaid
---
config:
   layout: elk
---
flowchart LR
   subgraph left["&nbsp"]
      direction TB

      subgraph ms[Mutable State]
         direction TB

         ms1["Workflow Task: Empty"] ~~~
         ms2["Activity Task: Scheduled, Attempt 2"]
      end

      subgraph transferQ[Transfer Queue]
         direction LR

         transfer1("&nbsp")
      end

      subgraph timerQ[Timer Queue]
         direction LR

         timer1("Activity Retry") ~~~
         timer2("Workflow Timeout")
      end

      ms ~~~ transferQ ~~~ timerQ
   end

   subgraph history[Workflow History]
      direction TB

      e1[WorkflowExecutionStarted] ~~~
      e2[WorkflowTaskScheduled] ~~~
      e3[WorkflowTaskStarted] ~~~
      e4[WorkflowTaskCompleted] ~~~
      e5[ActivityTaskScheduled]
   end

   left ~~~ history

   style left fill:transparent,stroke:transparent
<details> <summary><i>Code entrypoints</i></summary> </details>