Embracing the False Positive: Handling SSH Drops in Ansible

In the world of Infrastructure as Code, we are conditioned to treat every red line of text as a catastrophic failure. If Ansible prints FAILED!, something is broken, right?

Not always. Sometimes, an automation tool fails precisely because it succeeded too well.

Impact

• No actual outage: the workers powered down correctly every time.
• The safe-shutdown.sh script reported every worker as UNREACHABLE, which made a successful shutdown look like a failure and could stop a Bash script before it reached the next machine.

Timeline

1. I wrote a safe-shutdown.sh script to gracefully power off my bare-metal Kubernetes cluster.
2. After draining the workloads via the Kube API, the script triggered an Ansible ad-hoc command to power off the workers over SSH:

ansible workers -i hosts.yaml -m command -a "shutdown -h now"

3. The workers powered down, but the terminal filled with red errors:

fatal: [k8s-worker-01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh... Connection refused"}

4. I traced the error to a race condition between the shutdown and the SSH session, and treated it as an expected result.

Root cause

A race condition between the OS and the SSH daemon. When Ansible executes a command, it waits over the open SSH socket for an exit code (e.g., 0 for success).

shutdown -h now tells the Linux kernel to halt the system immediately. The machine powers off faster than Ansible can receive the success code. As the machine dies, it abruptly severs the SSH socket. Ansible sees an unexpectedly dropped connection and classifies it as a total failure.

Actions

Because the connection is expected to drop, the error is a false positive, and the scripts should embrace it.

When wrapping Ansible commands in Bash, append || true to suppress the failure and let the script continue to the next machine:

ansible workers -i hosts.yaml -m command -a "shutdown -h now" || true

In Ansible Playbooks, there are two cleaner options. Explicitly ignore the error:

- name: Shut down the node
  command: shutdown -h now
  ignore_errors: yes

Or, better, use the dedicated module, which is built to expect and handle the SSH disconnection gracefully:

- name: Shut down the node natively
  community.general.shutdown:

Lessons learned

• Not all errors are bugs: in automation, context is everything.
• When automation changes the state of the network or the power of the machine itself, "Connection refused" might just mean the command worked.
• Prefer modules designed for disruptive operations over generic command calls.