When a Pod dies inside a Kubernetes cluster, the event is never a single instantaneous failure but a coordinated sequence involving the kubelet, the container runtime, the API server, several independent controllers, and the networking layer, each of which reacts to the disappearance of a workload according to its own portion of the overall control loop. Treating Pod death as an abstract “self-healing” feature obscures the actual mechanics that determine termination latency, the possibility of dropped in-flight requests, and the precise guarantees Kubernetes does and does not provide during a failure. This walkthrough follows a single Pod from the moment it is declared unhealthy through termination, deregistration, and eventual rescheduling, examining each transition at the level of the specific component responsible for executing it.
Detection: How the Kubelet Determines That a Pod Has Failed
Failure detection begins locally, on the node where the Pod is running, and is the responsibility of the kubelet rather than the control plane. The kubelet continuously executes the liveness and readiness probes defined in the Pod specification, and a Pod is considered to have failed either when its liveness probe exceeds the configured failure threshold, when a container process exits unexpectedly and its restartPolicy does not permit an in-place restart, or when the node itself becomes unreachable and the node controller marks it as NotReady after the configured node-monitor-grace-period elapses. In the ordinary case of a liveness probe failure, the kubelet does not immediately involve the API server for the decision to terminate; it acts on local evidence first and reports the resulting state change afterward, which is why termination behavior can differ subtly between a container crash, a failed probe, and a node-level outage.
| Trigger | Detected By | Typical Latency |
|---|---|---|
| Liveness probe failure | Kubelet | Probe interval × failure threshold |
| Container process exit | Container runtime, reported via CRI | Near-immediate |
| Node becomes unreachable | Node controller | node-monitor-grace-period (default 40s) |
| Manual eviction / drain | API server, via eviction API | Immediate, subject to PodDisruptionBudget |
Termination: The SIGTERM, preStop, and SIGKILL Sequence
Once termination has been decided, the kubelet does not kill the container abruptly; it initiates a graceful shutdown sequence governed by the Pod’s terminationGracePeriodSeconds field, which defaults to thirty seconds when unspecified. The kubelet first invokes the container’s preStop lifecycle hook if one is defined, allowing the application to drain existing connections, deregister itself from external service discovery, or flush buffered state before the container receives any termination signal. Immediately after the preStop hook completes, or immediately if none exists, the kubelet instructs the container runtime through the Container Runtime Interface to deliver a SIGTERM to the container’s main process, at which point the application is expected to shut down on its own. If the process is still running when the grace period expires, the kubelet escalates and sends SIGKILL, which terminates the process unconditionally and does not permit further cleanup, making it critical that applications either honor SIGTERM promptly or that the grace period is tuned to match realistic shutdown duration.
apiVersion: v1
kind: Pod
metadata:
name: checkout-service
spec:
terminationGracePeriodSeconds: 45
containers:
- name: checkout-service
image: registry.example.com/checkout-service:1.4.2
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5 && /app/drain.sh"]
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 3The preStop hook above deliberately sleeps before invoking the drain script, a pattern that exists to close a well-known race condition: kube-proxy and the EndpointSlice controller remove a terminating Pod from Service routing asynchronously relative to the kubelet’s shutdown sequence, so a Pod that stops accepting connections the instant it receives SIGTERM can still receive a small number of requests routed by nodes that have not yet observed the endpoint removal.
Control Plane Reconciliation: How the Cluster Notices the Pod Is Gone
While termination proceeds on the node, a parallel reconciliation process unfolds in the control plane that is entirely independent of what the kubelet is doing to the container itself. The EndpointSlice controller watches Pod status through the API server and removes the terminating Pod’s IP address from the relevant Service’s endpoints, which is the mechanism that actually stops new traffic from being routed to it. Separately, the controller responsible for the owning workload, typically the ReplicaSet controller when the Pod belongs to a Deployment, observes through its watch on the API server that the number of Ready Pods matching its label selector has fallen below the desired replica count specified in the ReplicaSet’s spec. This discrepancy between desired and observed state is the entire trigger for recovery; the ReplicaSet controller does not know or care why the Pod disappeared, only that its reconciliation loop must create a replacement to restore the declared replica count.
Kubelet (node) API Server Controllers
───────────────── ────────── ───────────
Probe fails
→ preStop hook
→ SIGTERM
→ grace period
→ SIGKILL (if needed) → Pod status: Terminated → EndpointSlice controller
removes IP from Service
→ ReplicaSet controller
observed < desired
→ creates new Pod spec
→ Scheduler binds Pod to node
→ Kubelet on new node
pulls image, starts container
→ Readiness probe passes
→ EndpointSlice controller
adds new IP to ServiceRecovery: Scheduling, Startup, and Rejoining the Service
Recovery is executed by an entirely different set of components than detection and termination were, which is precisely why Kubernetes can restore capacity even when the original node has failed outright. The ReplicaSet controller’s newly created Pod object is picked up by the scheduler, which evaluates node affinity rules, resource requests, taints and tolerations, and topology spread constraints before binding the Pod to a suitable node; this binding decision is independent of which node hosted the previous instance, so a replacement Pod commonly lands on a different node entirely. The kubelet on the selected node then pulls the container image if it is not already cached, creates the container through the CRI, and begins evaluating the startup and liveness probes exactly as it did for the original Pod. Only once the readiness probe reports success does the EndpointSlice controller add the new Pod’s IP address back into the Service’s endpoints, meaning the interval between the original failure and full traffic restoration is the sum of scheduling latency, image pull time, container start time, and the readiness probe’s initial delay and success threshold, and this end-to-end recovery is inherently eventually consistent rather than instantaneous.
Common Mistakes That Undermine Recovery
- Missing or overly short terminationGracePeriodSeconds, which forces SIGKILL before the application has finished draining connections or persisting state.
- No preStop hook on Pods behind a Service, causing dropped requests during the window before the EndpointSlice controller propagates the removal.
- Absent or misconfigured readiness probes, which allows traffic to reach a replacement Pod before its dependencies, such as database connections or cache warm-up, are actually ready.
- No PodDisruptionBudget on availability-sensitive workloads, permitting voluntary disruptions such as node drains to remove too many Pods simultaneously.
- Treating restartPolicy as a substitute for a readiness probe, since a restarted container can pass its liveness probe while still being functionally unready to serve traffic.
Frequently Asked Questions
Does a new Pod reuse the same IP address as the Pod that died? No, a replacement Pod is a distinct object with its own IP address; identity-sensitive workloads that require stable network identity should use a StatefulSet rather than relying on Deployment-managed Pods.
Can Kubernetes guarantee zero dropped requests during Pod termination? No, because endpoint removal and container shutdown happen on independent timelines; minimizing dropped requests requires a preStop hook that outlasts the propagation delay of the EndpointSlice removal.
What happens if the node itself dies rather than just the Pod? The node controller waits for the node-monitor-grace-period before marking the node NotReady, after which the Pods on that node are marked for deletion and the same ReplicaSet reconciliation and scheduling process described above creates replacements on healthy nodes.
