Understanding Kubernetes Probes: A Deep Dive

Originally based on the ngrok blog post by Sam Rose

In this guide, we are going to explore exactly how Kubernetes probes function. The goal is to demonstrate how these mechanisms enhance application resilience and prevent common, avoidable pitfalls—such as the dreaded restart loops that take hours to resolve or the dropping of user requests during a rolling update.

To illustrate these concepts, I'm utilizing webernetes, a partial port of Kubernetes written in TypeScript. With over 100,000 lines of ported Go code, it allows for a simulated cluster to run directly in the browser. (Note: These behaviors have been cross-verified against k3s, and in the process, a bug in the actual Kubernetes codebase was discovered!)

🎯 Learning Objectives

• Understand the three distinct types of probes.
• Identify why pods without probes are risky.
• Learn how to configure startup probes to handle slow-starting apps.
• Distinguish between pod "Readiness" and actual traffic availability.

---

The Risk of a Pod Without Probes

Imagine we are running a simple pod with a single container. Here is the manifest for pod-a.yaml:

apiVersion: "v1"
kind: "Pod"
metadata:
  name: "pod-a"
spec:
  containers:
    - name: "app"
      image: "my-app:latest"

The my-app:latest image isn't instantaneous; it requires a few seconds of initialization before it can actually listen for traffic on port 8080.

The Crash Cycle

If the container crashes and is restarted by Kubernetes, two things happen:

1. Immediate Restart: The first crash triggers an instant restart.
2. Exponential Backoff: Subsequent crashes trigger a CrashLoopBackOff.

The delay for this backoff typically starts at 10 seconds and doubles with every failure, capping at a maximum of 5 minutes. This can be represented mathematically as: $\text{Wait Time} = \min(10 \times 2^n, 300) \text{ seconds}$ (where $n$ is the number of consecutive crashes).

The "Ready" Fallacy

The critical issue here is that Kubernetes considers the container Ready the moment the process starts, regardless of whether the application has finished its internal setup.

If we introduce pod-b (which acts as a client, like an Ingress controller or a Load Balancer) sending requests to pod-a every 2 seconds, we see a failure pattern:

Observation: From the moment the container restarts until the initialization logic completes, every request sent by pod-b will $\text{fail}$, even though Kubernetes reports the pod as Ready.

---

Enter: Kubernetes Probes

To solve this, we use probes: periodic health checks performed by the kubelet (the agent running on each node) to determine the state of a container.

The Three Flavors of Probes

---

Deep Dive: Startup Probes

Since our problem is a slow-starting application, the Startup Probe is the correct tool. Let's update pod-a.yaml:

apiVersion: "v1"
kind: "Pod"
metadata:
  name: "pod-a"
spec:
  containers:
    - name: "app"
      image: "my-app:latest"
      startupProbe:
        httpGet:
          path: "/startup"
          port: 8080
        periodSeconds: 1
        failureThreshold: 5

How it works

The kubelet sends a GET /startup request to port 8080. The configuration above creates a "grace window" for the app to start: $\text{Startup Window} = \text{periodSeconds} \times \text{failureThreshold} = 1\text{s} \times 5 = 5\text{s}$

If the probe fails 5 times consecutively, Kubernetes deems the startup a failure and kills the container.

Probing Methods:

• httpGet: Performs an HTTP request.
• tcpSocket: Checks if a TCP connection can be established.
• exec: Runs a specific command inside the container.
• grpc: Uses the gRPC health-checking protocol.

The State Transition

With a startup probe active, the pod does not immediately become Ready. It stays in a pending state until the first probe succeeds.

[!IMPORTANT] A Note on "NotReady" Technically, Kubernetes doesn't have a "NotReady" status. It has a Ready condition that can be True, False, or Unknown. In the demos, "NotReady" is used as shorthand for Ready=False.

---

The Gap Between Readiness and Routing

Even with a startup probe, if pod-b sends requests directly to pod-a's IP address, those requests will still fail during the startup window. This is because direct IP communication $\text{bypasses}$ the Kubernetes readiness mechanism.

To implement a production-grade solution, we must use a Service to load-balance traffic across multiple replicas.

Scaling with ReplicaSets and Services

We can define a ReplicaSet to ensure two copies of our pod are always running, and a Service to act as the stable entry point.

replica-set-a.yaml

apiVersion: "apps/v1"
kind: "ReplicaSet"
metadata:
  name: "replica-set-a"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: "pod-a"
  template:
    metadata:
      labels:
        app: "pod-a"
    spec:
      containers:
        - name: "app"
          image: "my-app:latest"
          startupProbe:
            httpGet:
              path: "/startup"
              port: 8080
            periodSeconds: 1
            failureThreshold: 5

service-a.yaml

apiVersion: "v1"
kind: "Service"
metadata:
  name: "service-a"
spec:
  selector:
    app: "pod-a"
  ports:
    - port: 80
      targetPort: 8080

Now, pod-b will send requests to service-a, which only routes traffic to pods that have successfully passed their probes.