Most SSH hardening guides stop at key-only auth and fail2ban. That helps, but if your machine has a public IP, port 22 is still open to the internet. Every automated scanner can probe it and get a response: the SSH version string, the banner, proof that something is listening.
The log noise alone is annoying: even with key-only auth, failed attempts pile up every day. But the bigger problem is exposure. If a zero-day drops in OpenSSH, every server with port 22 open is a target before you have time to patch. That has happened before.
I wanted the SSH daemon itself to be unreachable: no banner, no version string, nothing for nmap to work with. Not just rate-limited or hidden behind a non-standard port, but genuinely not connectable unless you already hold the key.
The idea: port knocking, and why it is not enough on its own
Port knocking is the original approach to this problem. The idea is simple: the server watches for a specific sequence of connection attempts on closed ports. If a client connects to port 7000, then 8000, then 9000, the server recognizes the pattern and temporarily opens a real port, like 22.
It works, but it has a real weakness: the knock sequence travels in the clear. Anyone watching your traffic can capture the sequence and replay it. There is no authentication, only obscurity.
fwknop solves this with Single Packet Authorization (SPA). Instead of a knock sequence, you send a single UDP packet that is encrypted and cryptographically signed with an HMAC. The server only opens port 22 if it can verify the packet came from someone holding the right keys.
How Single Packet Authorization works
Before SSH is accessible at all, you send one encrypted UDP packet to the server. fwknopd validates the packet and temporarily inserts a firewall rule that opens port 22 for your source IP only, for a configurable time window (I use 120 seconds). After that window, the rule is removed automatically. If you are already connected, the session stays up because the connection was established before the rule disappeared.
From a scanner’s perspective, port 22 never responds. nmap reports it as filtered, the same state as any silently dropped packet. No banner, no RST, no confirmation that SSH is even listening there.
Because SPA uses UDP, the packet can be lost in transit. If SSH hangs on connect, run fwknop -n server1 again and retry. It is stateless, so resending is safe.
Left: a port scanner gets no response. Right: a valid SPA packet opens port 22 for the client’s IP for 120 seconds, then closes it again.
The SPA packet itself is HMAC-SHA512-authenticated and encrypted. Replaying a captured packet does not work: the packet contains a timestamp and random data, and fwknopd keeps a digest of every packet it has already accepted. It checks both, and drops the packet silently if the packet is too old or its digest has been seen before.
Generating keys
You need two keys: an encryption key and an HMAC key. The encryption key protects the contents of the SPA packet. The HMAC key authenticates it. They serve different purposes and should be separate; using the same key for both weakens the security properties of the scheme.
Key generation is a one-time step, and you need the output before configuring either side:
fwknop --key-genThis outputs something like:
KEY_BASE64: <long-base64-string>
HMAC_KEY_BASE64: <long-base64-string>Store both keys somewhere you can retrieve them later: you will need them on the server (via Ansible) and on every client machine. I keep them in a SOPS-encrypted Ansible vars file, group_vars/linux/vault.sops.yaml, as vault_fwknop_spa_key and vault_fwknop_hmac_key, and mirrored in Bitwarden as two login items, fwknop_spa_key and fwknop_hmac_key, with each key stored in the password field. The two stores serve different tools: Ansible reads the SOPS file when deploying the server side, while chezmoi queries Bitwarden when populating ~/.fwknoprc on client machines. Same keys, two entry points.
One key pair covers all hosts. Each host runs the same fwknopd configuration pointing to the same shared keys. If you want per-host keys you can use separate stanzas in the access file, but one shared pair is simpler to rotate.
Deploying the server side with Ansible
My Ansible role for this is fwknop_server. It installs fwknop-server, deploys the server configuration and access file, and opens the SPA port in UFW.
The key part of the tasks file:
- name: Install fwknop-server
apt:
name: fwknop-server
state: present
- name: Deploy fwknopd.conf
template:
src: fwknopd.conf.j2
dest: /etc/fwknop/fwknopd.conf
owner: root
group: root
mode: "0600"
- name: Deploy access.conf
template:
src: access.conf.j2
dest: /etc/fwknop/access.conf
owner: root
group: root
mode: "0600"
no_log: true
- name: Allow fwknop SPA port (UDP)
ufw:
rule: allow
port: "{{ fwknop_spa_port | string }}"
proto: udp
comment: "fwknop SPA"The public SSH UFW rule is removed by my base role on every host with fwknop enabled (more on that below). From that point, iptables only opens port 22 when a valid SPA packet arrives. fwknopd does this by inserting iptables rules directly, beneath UFW’s management layer, so the temporary rules are invisible to ufw status.
The defaults:
fwknop_spa_port: 62201
fwknop_access_timeout: 120 # seconds
fwknop_ssh_port: "{{ ssh_port | default(22) }}"fwknop_access_timeout flows into access.conf.j2 as the FW_ACCESS_TIMEOUT value, which tells fwknopd how long to keep the firewall rule open after a valid SPA packet. (ACCESS_EXPIRE is a different directive: it sets when the stanza stops accepting SPA packets altogether.)
The client configuration
The client side is ~/.fwknoprc, which chezmoi manages and populates with keys from Bitwarden at apply time. The file has a stanza per host:
[default]
SPA_SERVER_PROTO udp
HMAC_DIGEST_TYPE SHA512
USE_HMAC Y
ALLOW_IP resolve
ACCESS tcp/22
KEY_BASE64 {{ (bitwarden "item" "fwknop_spa_key").login.password }}
HMAC_KEY_BASE64 {{ (bitwarden "item" "fwknop_hmac_key").login.password }}
[server1]
SPA_SERVER 203.0.113.10
SPA_SERVER_PORT 62201
[server2]
SPA_SERVER server2.example.com
SPA_SERVER_PORT 62201ALLOW_IP resolve tells fwknop to detect your current public IP automatically. The keys are base64-encoded and come from the generation step above.
One trap on a new laptop: installing fwknop drops a sample ~/.fwknoprc in your home directory, with every line commented out. My template keeps an existing file when the password manager is locked, so chezmoi apply still works offline, and at first it accepted any non-empty file as that cached copy. The sample file won. fwknop -n server1 failed with “stanza not found”, and since the Ansible setup below throws away fwknop’s errors, all I saw was SSH timing out. The fallback now keeps only the two keys, read from the [default] stanza of the deployed file and only when they look real. The host stanzas always come from the template, so a new host lands even with the vault locked, and a commented-out sample file yields placeholder keys that a check after the apply reports.
Wiring it into ~/.ssh/config
The knock can be made fully transparent by adding a ProxyCommand to each host in ~/.ssh/config. Once this is in place, ssh server1 just works: the SPA packet goes out, the port opens, SSH connects.
Host server1
HostName 203.0.113.10
ProxyCommand fwknop -n server1; sleep 2; nc %h %p
Host server2
HostName server2.example.com
ProxyCommand fwknop -n server2; sleep 2; nc %h %pfwknop -n server1 selects the [server1] stanza from ~/.fwknoprc to know where to send the SPA packet. The sleep 2 gives fwknopd time to update iptables before nc (netcat) opens the actual TCP connection to the now-unblocked port.
Wiring it into Ansible
Initial provisioning of a new machine uses plain SSH: fwknop is not installed yet, so the connection is direct. Once the fwknop_server role runs and the UFW rule is removed, every subsequent Ansible run needs to knock first.
I handle this by setting ansible_ssh_common_args in group_vars/linux/vars.yaml, only for hosts that set fwknop_enabled; the others connect directly, without the knock or the two-second wait:
fwknop_ssh_proxy_args: "-o ProxyCommand=\"sh -c 'fwknop -n {{ inventory_hostname }} 2>/dev/null; sleep 2; exec nc %h %p'\""
ansible_ssh_common_args: "{{ fwknop_ssh_proxy_args if fwknop_enabled | default(false) | bool else '' }}"The stanza name comes from inventory_hostname, not from SSH’s %h: %h is whatever ansible_host resolves to, which can be an IP address, while the stanzas in ~/.fwknoprc are named after the inventory hosts. Errors from fwknop are discarded on purpose. On a fresh machine port 22 is still open, the knock fails silently and nc connects directly, so the same setting works before and after fwknop_server has run. With SSH connection reuse turned on in ansible.cfg, the knock goes out once per play rather than once per task.
The rule that kept coming back
Months after setting this up, I found port 22 open to the internet again on my cloud VM. Nothing had failed: every Ansible run finished green. Two of my roles were simply undoing each other.
My base role, linux_base, adds a rate-limited public SSH rule unless a host sets linux_base_ssh_public: false. fwknop_server deleted that same rule. Each run went like this: fwknop_server removed the rule, and then a later role that lists linux_base as a role dependency ran it again, which put the rule back. The last writer wins, and the last writer was the wrong one.
Ordering the roles differently would only hide the problem until the next role with that dependency. The fix is to make both roles agree on what the rule should be, so it no longer matters which one runs last:
# linux_base defaults
linux_base_ssh_public: "{{ not (fwknop_enabled | default(false) | bool) }}"On a host with fwknop, linux_base now takes its “remove public SSH” path too, and a second run changes nothing. If two roles touch the same piece of state, they need to read the same variable to decide what it should be; otherwise the result depends on play order.
Later I removed the cause as well: no role lists linux_base as a dependency any more, because it already runs first on every Linux host. And fwknop_server no longer deletes the rule at all: linux_base owns it, and the shared variable decides whether it exists.
Tailscale as a parallel access path
Alongside fwknop, the Tailscale interface (tailscale0) always permits SSH with no knock required. In fact UFW accepts all traffic on that interface: who can reach tailscale0 at all is decided by the tailnet’s access rules, not by UFW. This is not a fallback for when fwknop breaks; it is a separate access path that exists independently.
In practice it covers one scenario well: if a configuration mistake locks out the public interface, any Tailscale-connected device can still reach the machine over the tailnet.
# UFW rule added by the tailscale role
ufw allow in on tailscale0What this actually changes
Before: port 22 open, fail2ban watching for brute-force attempts, log noise every day. A zero-day in OpenSSH is a real problem you have to respond to immediately.
After: port 22 is invisible to the internet. No banner to grab. No connection attempts in auth logs because there is nothing to connect to. The attack surface shrinks to the fwknop UDP port, which drops anything that does not validate as a properly signed SPA packet. A zero-day in OpenSSH still matters, but the exposure window is different: an attacker cannot even reach the SSH daemon without the key.
This is not security through obscurity. The port is not hidden behind some clever port number. It is genuinely closed. The SPA packet is what proves you belong there, the same way a key proves you belong inside a locked room. The obscurity is just a side effect. And from the outside, ssh server1 still works as it always did: the knock is invisible inside the ProxyCommand.