CRUD.cloud

The Read-Only Lockdown: Recovering a Bare-Metal Node from a Kernel Panic

What do you do when your automated deployment pipeline throws an UNREACHABLE error on a machine that you can successfully SSH into? This is the exact scenario I faced recently when bringing up a third worker node in my bare-metal Kubernetes cluster.

Impact

  • k8s-worker-01 (an older Dell laptop repurposed as a worker node) could not be configured by Ansible and was unusable as a Kubernetes node.
  • The node needed a hard power cycle and a manual filesystem repair before it could rejoin the cluster.

Timeline

  1. I ran my standard suite of Ansible playbooks to standardize the configuration across the cluster.
  2. On k8s-worker-01, Ansible threw a confusing error:
UNREACHABLE! Failed to create temporary directory
  1. This usually means a network drop or SSH daemon failure, but a manual SSH into the node worked: I had a prompt.
  2. A simple file creation test:
touch ~/test

failed with:

touch: cannot touch '/home/leva/test': Read-only file system
  1. dmesg | tail -n 30 showed a string of EXT4-fs error messages: the root filesystem (/) had been locked read-only.
  2. I tried to reboot:
sudo reboot

The command hung indefinitely, leaving the machine in a zombie state. 7. I did a physical hard power cycle, holding the power button until the machine died. 8. On boot, the kernel's automated fsck could not repair the ext4 journal: the drive was still mounted read-only. 9. I ran a manual, aggressive filesystem check on the unmounted partition (see Actions). After several minutes of fixing orphaned inodes and corrupted blocks, fsck reported the drive was clean. 10. After remounting read-write, the node rejoined the Kubernetes cluster and containerd came back online.

Root cause

The older SSD suffered a hardware-level fault or a kernel memory panic. To protect the integrity of the data and prevent further corruption, the Linux kernel deliberately remounted the entire drive as read-only. Ansible could not create its temporary directory, hence the misleading UNREACHABLE.

The reboot failed for the same reason: with a read-only filesystem, systemd could not write the shutdown logs or update the runlevel states required to gracefully halt the OS.

Actions

  1. Hard power cycle the node (graceful shutdown is impossible on a read-only root).
  2. Force a manual filesystem check on the unmounted partition:
sudo fsck -y /dev/sda1
  1. Remount the root filesystem read-write with sudo mount -o remount,rw /.
  2. Check that the node rejoins the cluster and containerd is running.

Lessons learned

  • In the cloud, hardware failures are abstracted away: a dead VM is terminated and replaced by an Auto Scaling Group. On bare metal, you are the final line of defense against failing SSDs and kernel panics.
  • Always check dmesg when weird I/O or "temporary directory" errors occur.
  • When the filesystem locks, even standard shutdown commands will fail you: plan for physical access or out-of-band power control.