One host in a fleet run wedged the entire play for nine minutes before I gave up and killed it. The host wasn’t down — it answered pings, and ssh host-b true returned instantly. But the moment Ansible tried to do real work on it, everything froze. And with serial: 1, one stuck host means the whole run is stuck.

Here is what is actually happening, and how to make it impossible.

The failure mode

The host was in a half-up state: reachable at the TCP layer, and the SSH handshake completed — you even get the friendly Warning: Permanently added 'host-b' to the list of known hosts — but the actual command and module execution hung. This shows up when an SSH daemon accepts connections but can’t spawn a session, or, in my case, when the control connection runs over a mesh VPN tunnel that dropped mid-session: the socket is ESTABLISHED but silently black-holing packets.

Your instinct is “surely there is a timeout for this.” There is. It just does not cover this case:

• ansible_timeout / ConnectTimeout bounds the TCP connect only. The connect already succeeded, so this never fires.
• ServerAliveInterval / ServerAliveCountMax only begin counting after the session is established. A hang during key exchange or command exec sits before or around that window, so they do not reliably kick in.
• wait_for_connection’s timeout feels like the answer, but it is checked between connection attempts. If one attempt hangs, the loop is stuck inside that attempt and never gets to evaluate the timeout.

So nothing at the Ansible layer bounds a hung ssh invocation. The bound has to come from outside the ssh process.

The fix: wrap ssh in timeout

Point Ansible at a wrapper that runs ssh under coreutils timeout, so no single ssh call — connect, handshake, auth, or module transfer — can outlive it:

- name: Fleet run
  hosts: all
  become: true
  serial: 1
  gather_facts: false
  ignore_unreachable: true
  vars:
    ansible_timeout: 30
    ssh_timeout_wrapper: /tmp/ssh-timeout.sh
    ansible_ssh_executable: "{{ ssh_timeout_wrapper }}"
    # ControlMaster=no keeps each ssh a standalone process the timeout can cleanly kill
    ansible_ssh_common_args: >-
      -o ServerAliveInterval=10 -o ServerAliveCountMax=3
      -o ControlMaster=no -o ControlPath=none
  pre_tasks:
    - name: Install the timeout-wrapped ssh on the controller
      delegate_to: localhost
      become: false
      ansible.builtin.copy:
        dest: "{{ ssh_timeout_wrapper }}"
        mode: "0755"
        content: |
          #!/bin/sh
          exec timeout -k 5 60 ssh "$@"

That is the whole trick. ansible_ssh_executable tells Ansible to invoke /tmp/ssh-timeout.sh instead of ssh; the wrapper just re-execs ssh under timeout -k 5 60 — SIGTERM at 60 seconds, SIGKILL five seconds after that if it is really stuck. A wedged host now fails its task within about a minute instead of hanging forever. Pair it with ignore_unreachable: true and a block/rescue around your tasks, and the run logs the bad host and moves on instead of dying.

Details that bite

• Set the timeout above your slowest legitimate synchronous task. An apt update against a slow mirror can hold one SSH session for 30–40 seconds. 60 is a safe default; 30 will start killing real work.
• Disable ControlMaster. Ansible’s default SSH multiplexing keeps a persistent master connection alive. If timeout kills that master mid-life, other multiplexed sessions break in confusing ways. ControlMaster=no makes each ssh a clean, independently killable process. You give up a little speed and get determinism.
• Pipelining helps here. With pipelining = True in ansible.cfg, the module is piped over that single wrapped ssh, so the module transfer is bounded too — no separate sftp/scp connection sneaking past the timeout.
• The wrapper lives on the controller. delegate_to: localhost writes it wherever Ansible runs — your laptop, a CI runner, a Semaphore box — never on the targets.

But shouldn’t you just fix the host?

Yes — a host that hangs SSH is broken and deserves attention. But fleet automation should never be one bad node away from a nine-minute hang, let alone an indefinite one. This wrapper converts “the whole run is stuck” into “one host got skipped, and here is which one” — which is exactly the failure mode you want at 3 a.m.