CRUD.cloud

When Kubelet Dies Silently: Diagnosing Container Runtime Failures

One of the most frustrating alerts in Kubernetes operations is seeing a node drop to a NotReady or NodeStatusUnknown state. When you run kubectl describe node, the only clue you get is:

Kubelet stopped posting node status.

Where do you even begin to look when the primary node agent dies silently?

This post gathers several incidents from my bare-metal homelab, from read-only filesystem lockdowns to misconfigured network plugins, that all ended the same way.

Impact

  • The node drops to NotReady / NodeStatusUnknown and stops running workloads.
  • Pods on it are stuck (for example in ContainerCreating), and the only visible symptom is Kubelet stopped posting node status.

Timeline

The pattern seen across these incidents:

  1. A lower layer fails: the disk goes read-only, or the CNI plugins can't be found.
  2. containerd crashes or locks up, and its CRI socket disappears.
  3. The kubelet loses its socket and crash-loops.
  4. The node stops posting status and drops to NotReady.

In the CNI case specifically: I deployed the Flannel Container Network Interface (CNI), pods refused to leave the ContainerCreating state, and shortly after the node dropped offline entirely.

Root cause

The Kubelet does not exist in a vacuum. The kubelet depends entirely on its Container Runtime Interface (CRI) socket. In modern setups this is usually containerd, listening at /var/run/containerd/containerd.sock. If the kubelet cannot communicate with that socket, it doesn't just log an error and wait: it intentionally crash-loops.

The two most common reasons containerd was dead:

  1. The hardware lock. When the node's SSD suffers a panic and remounts as read-only, containerd crashes immediately because it can no longer write container logs or manage overlay filesystems. When containerd dies, the CRI socket disappears, and kubelet follows it to the grave.
  2. The CNI path mismatch. Debian's containerd package expects CNI plugins to live in /usr/lib/cni/, but the Flannel installer put them in the standard /opt/cni/bin/. containerd couldn't initialize the network, locked up, and dragged the kubelet down with it.

Actions

  1. When SSH'd into a node with a dead kubelet, don't start with systemctl restart kubelet. Look down the stack first:
sudo systemctl status containerd
  1. If containerd is dead because the disk went read-only, recover the filesystem first (see The Read-Only Lockdown).
  2. For the CNI path mismatch, a symbolic link fixed the entire chain:
sudo ln -sf /opt/cni/bin/* /usr/lib/cni/

Lessons learned

  • Kubernetes is a layered architecture. When a high-level agent like the kubelet stops reporting, don't just restart it: look at the foundation it stands on.
  • If the container runtime is dead, the node is dead. Fix the runtime, and the kubelet recovers automatically.