Updated 2026: The infrastructure described here (Kubernetes + Terraform + Salt on Hetzner) has since been replaced. This site now runs on Hugo deployed to Cloudflare Pages. The lessons about Phoenix Servers and zero-downtime DNS switching still apply; I’ve just moved to a simpler stack that needs less operational overhead. The full migration story is in From WordPress to Hugo and Cloudflare Pages.

I switched my entire cloud infrastructure from DigitalOcean to Hetzner with zero downtime. If you’re reading this, it worked.

The Phoenix Server concept

The key to zero-downtime infrastructure migration is the Phoenix Server pattern: at any point, you can recreate the entire infrastructure from scratch in an automated way. No manual steps, no tribal knowledge required.

My infrastructure was built in three layers:

  • Resource provisioning via Terraform: the Hetzner provider let me terraform apply the entire infrastructure (cloud instances, networking, DNS) from a declarative config.
  • Configuration management via Salt, versioned in Git: applying the Salt highstate on a fresh instance reproduced my exact configuration (Fail2ban rules, SSH hardening, everything).
  • Application stack on Kubernetes: every service runs in a container, orchestrated by Kubernetes, portable between providers.

I invested significant effort upfront in encoding all my configuration tweaks as Salt states. That investment paid back here: spinning up a complete, identical infrastructure on Hetzner was a terraform apply + salt highstate away.

The switch

With a complete Hetzner environment running in parallel, the old DigitalOcean infrastructure was still serving traffic. To test the new one, I edited /etc/hosts to point the domain to the new reverse proxy IP, bypassing DNS, hitting the new infrastructure directly. It worked correctly.

The actual cutover was a single DNS change: I updated the CNAME for the www record from the old reverse proxy to the new one. As DNS propagated, traffic shifted to Hetzner. I watched the reverse proxy logs in real time; as soon as upstream resolvers picked up the new record, requests started coming in through Hetzner.

Lessons on DNS during migrations

A few things worth noting for future reference:

  • A DNS wildcard record (*.michelebologna.net) resolving to a hostname can produce unexpected results if any machine has search michelebologna.net in its /etc/resolv.conf: short hostnames get expanded and may resolve unexpectedly.
  • Reverse DNS matters for mail: set PTR records for your mail server IPs to match the Postfix myhostname, and make sure DKIM, SPF, and DMARC records are in place before switching MX records.
  • The root record (@) must be an A/AAAA record, not a CNAME. CNAME at the apex is forbidden by RFC 1912.

Goodbye to the old reverse proxy

After draining traffic from DigitalOcean and verifying everything was running cleanly on Hetzner, I shut down the old infrastructure. The reverse proxy had been running for 5.5 years with 99.954% availability:

text
     #               Uptime | System
----------------------------+-----------------------
     1   112 days, 18:33:34 | Linux 4.4.0-tuned
     2   104 days, 21:00:22 | Linux 4.15.0-generic
     3    85 days, 19:08:32 | Linux 3.13.0-generic
     4    78 days, 19:04:49 | Linux 4.4.0-tuned
     5    71 days, 13:01:09 | Linux 4.13.0-lowlaten
     6    66 days, 04:42:44 | Linux 4.15.0-generic
     7    62 days, 15:49:14 | Linux 3.19.0-generic
     8    62 days, 00:52:09 | Linux 4.15.0-generic
     9    56 days, 22:21:20 | Linux 3.19.0-generic
    10    53 days, 16:34:11 | Linux 4.2.0-highmem
----------------------------+-----------------------
    up  1989 days, 03:46:34 | since Tue Oct 28 14:28:05 2014
  down     0 days, 22:00:33 | since Tue Oct 28 14:28:05 2014
   %up               99.954 | since Tue Oct 28 14:28:05 2014

Not bad for a machine I set up and largely forgot about.