CRUD.cloud

The Chicken-and-Egg Deadlock: Bootstrapping Kubernetes 1.29+ with kube-vip

Bootstrapping a highly available Kubernetes cluster on bare-metal usually involves setting up a Virtual IP (VIP) that floats between control plane nodes. My tool of choice for this is kube-vip. But recently, while bootstrapping a fresh Debian 13 cluster on Kubernetes 1.29+, I ran into a complete deadlock.

Impact

  • kubeadm init hung forever: the new highly available control plane never came up.
  • The cluster bootstrap was blocked until the VIP assignment was decoupled from kube-vip.

Timeline

  1. Following the standard kube-vip procedure, I generated a static pod manifest in /etc/kubernetes/manifests/ and ran kubeadm init. The idea is that as kubeadm brings up the API server, the kube-vip static pod spins up alongside it and claims the VIP (192.168.1.50).
  2. kubeadm init hung, waiting for the control plane.
  3. The kube-vip logs repeated this error:
leases.coordination.k8s.io "plndr-svcs-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases"
  1. A 403 Forbidden for the admin on a newly bootstrapping cluster pointed to an RBAC change in Kubernetes 1.29.
  2. I bound the VIP at the OS level, re-ran kubeadm init, and only then deployed kube-vip.

Root cause

Kubernetes 1.29 introduced a security change to how kubeadm generates the admin.conf file used during initialization. Historically, kubeadm granted the bootstrapping admin the system:masters superuser group, which bypasses all RBAC (Role-Based Access Control) checks. In 1.29+, that bypass was removed: kubeadm now relies entirely on standard RBAC roles to grant permissions.

This creates a chicken-and-egg problem:

  1. kubeadm starts the API server and tries to talk to it via the VIP (192.168.1.50).
  2. kube-vip tries to claim the VIP so kubeadm can reach it.
  3. To claim the VIP, kube-vip needs to write a coordination lease to the API server.
  4. Because the cluster is actively bootstrapping, the RBAC rules haven't been fully populated yet.
  5. Because RBAC isn't ready, and system:masters is gone, kube-vip is denied permission to create the lease (403 Forbidden).
  6. Because kube-vip fails, the VIP is never claimed.
  7. Because the VIP is never claimed, kubeadm can't reach the API server, and hangs forever.

Actions

Separate the VIP assignment from the Kubernetes static pod lifecycle.

  1. Before running kubeadm init, bind the VIP to the network interface at the OS level:
sudo ip addr add 192.168.1.50/24 dev eth0
  1. Run kubeadm init. It talks to the API server over the VIP (handled by the OS), finishes bootstrapping, and fully populates the RBAC rules.
  2. Once kubeadm init has completed, generate the kube-vip static pod manifest. When kube-vip boots, the RBAC rules exist, it writes its lease, and it takes over management of the VIP from the OS.

Lessons learned

  • Security improvements often break established workflows in unexpected ways: read the release notes of a new Kubernetes minor version before bootstrapping with it.
  • When a static pod throws a 403 during bootstrap, remember that in modern Kubernetes RBAC is king, and it isn't ready until the cluster is.