docs/recipes/probes.md
Just like in a Kubernetes deployment, you can define liveness and readiness ports in Tye, as part of the service.
Probes serve two main functions
Before we get to the definition of a probe and how it works, it's important to understand the life cycle of a replica.
When using the local orchestrator (tye run), each replica in Tye can be in one of these three states
Started - A replica is in this state when it has just been started, and hasn't been probed yet.Healthy - A replica is in this state when it passes the liveness probe, but not the readiness probe.Ready - A replica is in this state when it passes both the liveness and readiness probes.(Internally, there are more states that a replica can be in, but those are not relevant to this discussion)
The orchestrator is responsible for switching between the different states based on the feedback from the probes, as described in later sections.
When deploying the application to Kubernetes (via tye deploy), the life cycle is represented differently and is managed by Kubernetes. Click here to read more about the life cycle of a Pod in Kubernetes.
Throughout the rest of the document, we'll be referring to the life cycle of the replica as it is represented in the local orchestrator (i.e. Started, Healthy, Ready).
There are two types of probes, both have a similar schema but serve a different purpose.
The liveness probe is used to let Tye know when it can restart a replica.
When a replica is restarted due to a failed liveness probe, a new replica is created in its stead.
The readiness probe is used to let Tye know when it's okay to route traffic to a replica.
The readiness probe cannot kill/restart a replica, it can only do two things
Healthy replica to Ready, when the probe succeeds.Ready replica to Healthy when the probe fails.Tye only routes traffic (either via a service binding, or an ingress) to Ready replicas.
By now, the life cycle of a replica when both liveness and readiness probes are configured should be clear, but how does the life cycle of a replica look like when both probes or one of the probes is absent?
liveness nor readiness probes are present, a replica gets promoted from Started to Ready automatically, upon creation.liveness probe is present, a replica gets promoted from Started to Healthy only after it passes the liveness probe, but upon being promoted to Healthy, it's automatically promoted again, to Ready.readiness probe is present, a replica gets promoted from Started to Healthy automatically, upon creation, but gets promoted from Healthy to Ready only upon passing the readiness probe.The section will refer to this sample application which demonstrates basic usage.
To run the sample locally, clone this repository or download the source and navigate to the /samples/liveness-and-readiness/ directory in your terminal.
tye run
The sample has a single service with a liveness and a readiness probe, as shown in this snippet
services:
- name: simple-webapi
project: webapi/webapi.csproj
replicas: 2
liveness:
http:
path: /lively
readiness:
http:
path: /ready
After executing tye run, you should see these log lines
[18:14:05 INF] Replica simple-webapi_a2d67bd9-4 is moving to an healthy state
[18:14:05 INF] Replica simple-webapi_a9c2e2f4-d is moving to an healthy state
[18:14:07 INF] Replica simple-webapi_a9c2e2f4-d is moving to a ready state
[18:14:07 INF] Replica simple-webapi_a2d67bd9-4 is moving to a ready state
As you can see, both replicas pass the liveness probe and get prompted to an Healthy state, and shortly after, both replica pass the readiness probe and get promoted to a Ready state.
If the readiness probe fails, a replica will move from a Ready state to an Healthy state.
[18:14:18 INF] Replica simple-webapi_a2d67bd9-4 is moving to an healthy state
If after some time, the readiness probe becomes successful again, a replica will move from an Healthy state to a Ready state.
[18:14:26 INF] Replica simple-webapi_a2d67bd9-4 is moving to a ready state
If the liveness probe fails, a replica will be killed, and the orchestrator will spawn a new replica in its stead
[18:25:08 INF] Killing replica simple-webapi_a9c2e2f4-d because it has failed the liveness probe
...
[18:25:08 INF] Launching service simple-webapi_0e7fe12d-7
[18:25:09 INF] Replica simple-webapi_0e7fe12d-7 is moving to an healthy state
[18:25:11 INF] Replica simple-webapi_0e7fe12d-7 is moving to a ready state
When you deploy an application with liveness and/or readiness probes to Kubernetes, these probes get translated to their equivalent representation in Kubernetes
After running
tye deploy --interactive
You should see the simple-webapi deployment in Kubernetes
NAME READY UP-TO-DATE AVAILABLE AGE
simple-webapi 2/2 2 2 90s
Run
kubectl describe deploy simple-webapi
And you will notice that the deployment has Liveness and Readiness in its description
Liveness: http-get http://:80/lively delay=0s timeout=1s period=1s #success=1 #failure=3
Readiness: http-get http://:80/ready delay=0s timeout=1s period=1s #success=1 #failure=3