Why We Swapped NGINX for Caddy Across Our Fleet

For over a decade, reaching for NGINX was an automatic reflex whenever I spun up a new Linux server or designed a container ingress layer. It was fast, battle-tested, and ubiquitous across the industry, which meant nobody ever questioned the operational tax required to keep it running safely in production. We accepted that deploying a new web service meant copying eighty lines of arcane proxy headers from an older project, mounting shared volumes for Let's Encrypt challenges, and wiring up a fragile Certbot cron daemon that silently prayed nothing would block port eighty during renewal windows. It took a preventable Saturday night certificate expiration outage across one of our multi-tenant platforms to make me ask why we were still manually babysitting transport layer security in the modern cloud era.

The Hidden Fragility of Boilerplate Proxy Configurations

When we audited our existing ingress containers after that weekend incident, we realized how much defensive boilerplate we had accumulated just to make NGINX behave like a modern web server. Every single service definition required explicit directives to upgrade WebSocket connections, strip server tokens, configure modern TLS cipher suites, enable Gzip compression, and forward client IP headers accurately to upstream Node.js and Go containers. Worse still, coupling NGINX with external ACME scripts meant our reverse proxy and our certificate lifecycle lived in two completely separate processes that only communicated by poking files on disk and firing system reload signals.

1# The legacy NGINX boilerplate we maintained across dozens of services
2server {
3    listen 443 ssl http2;
4    server_name api.ourplatform.com;
5
6    ssl_certificate /etc/letsencrypt/live/api.ourplatform.com/fullchain.pem;
7    ssl_certificate_key /etc/letsencrypt/live/api.ourplatform.com/privkey.pem;
8    ssl_protocols TLSv1.2 TLSv1.3;
9
10    location / {
11        proxy_pass http://upstream_app:3000;
12        proxy_http_version 1.1;
13        proxy_set_header Upgrade $http_upgrade;
14        proxy_set_header Connection "upgrade";
15        proxy_set_header Host $host;
16        proxy_set_header X-Real-IP $remote_addr;
17        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
18        proxy_set_header X-Forwarded-Proto $scheme;
19    }
20}

Replacing that entire setup with Caddy felt almost suspiciously simple during our initial proof of concept. Because Caddy was engineered from the ground up in Go with automatic HTTPS as a core primitive rather than an afterthought plugin, ninety percent of our configuration files simply vanished. By default, Caddy provisions certificates from Let's Encrypt or ZeroSSL, renews them automatically in the background, redirects HTTP to HTTPS, enables HTTP/3 over QUIC, and forwards all standard proxy and WebSocket headers without requiring a single extra line of instruction.

Solving Multi-Tenant Custom Domains With On-Demand TLS

While shrinking static configuration files was a welcome relief, the real architectural breakthrough happened when we migrated our multi-tenant storefront engine. In our business model, merchants and partner agencies frequently map their own custom vanity domains to our platform via CNAME records. Under our legacy NGINX architecture, onboarding a new customer domain required a complex background queue that generated a temporary virtual host file, executed a Certbot shell command, waited for DNS propagation, and issued a reload signal to the master NGINX process.

Caddy eliminated that entire orchestration pipeline through a native feature called On-Demand TLS. Instead of provisioning certificates ahead of time via background workers, Caddy can dynamically negotiate and cache a brand-new TLS certificate right during the very first TLS handshake when a visitor opens a customer's custom domain in their browser. To prevent malicious actors from pointing random domains at our load balancer and exhausting our rate limits, we simply pointed Caddy's internal verification hook at a lightweight endpoint on our core backend.

1{
2	on_demand_tls {
3		ask http://internal-core-api:4000/api/internal/verify-domain
4		interval 2m
5		burst 5
6	}
7}
8
9https:// {
10	tls {
11		on_demand
12	}
13
14	encode zstd gzip
15
16	reverse_proxy upstream_storefront:3000 {
17		health_uri /api/health
18		health_interval 10s
19		health_timeout 3s
20	}
21}

Whenever an unknown domain hits port 443 across our fleet, Caddy pauses the handshake for a few milliseconds and fires an internal HTTP request to our verification route with the requested domain name as a query parameter. Our backend checks PostgreSQL or Redis to confirm whether that domain belongs to an active paying tenant and returns a simple status code before Caddy ever contacts a certificate authority.

1import { NextRequest, NextResponse } from "next/server"
2import { db } from "@/db"
3import { tenantDomains } from "@/db/schema"
4import { eq, and } from "drizzle-orm"
5
6export async function GET(request: NextRequest) {
7  const domain = request.nextUrl.searchParams.get("domain")
8
9  if (!domain) {
10    return new NextResponse("Missing domain parameter", { status: 400 })
11  }
12
13  const activeRecord = await db.query.tenantDomains.findFirst({
14    where: and(
15      eq(tenantDomains.customDomain, domain.toLowerCase()),
16      eq(tenantDomains.isVerified, true)
17    ),
18  })
19
20  if (!activeRecord) {
21    return new NextResponse("Unauthorized custom domain", { status: 403 })
22  }
23
24  return new NextResponse("Domain verified for TLS issuance", { status: 200 })
25}

This twelve-line verification handler completely replaced over six hundred lines of brittle shell scripting and job queue retry logic. When a customer configures their DNS record today, their custom domain goes live with full TLS 1.3 encryption and Zstandard compression the exact second their first visitor loads the page. Furthermore, by configuring Caddy's distributed storage module to point at our shared Redis cluster or S3-compatible bucket, every Caddy replica across our fleet automatically shares the issued certificates without duplicate ACME challenges.

Zero-Reload Routing Updates Under Heavy WebSocket Load

Another subtle victory emerged once we looked closely at our long-lived connection metrics. Whenever NGINX receives a configuration reload signal to apply upstream changes or rotated certificates, it spawns a brand-new set of worker processes and marks the old workers for shutdown. When you run high-frequency WebSocket streams or server-sent events for real-time order tracking, frequent configuration reloads leave lingering zombie worker processes hanging in memory for hours until every client finally disconnects.

Caddy avoids process-forking gymnastics entirely because its core engine is controlled through an in-memory JSON administration API. Even when you write human-friendly Caddyfiles, Caddy compiles them into a structured JSON document under the hood and applies incremental state diffs in memory without restarting worker threads or dropping active TCP sockets. Our deployment orchestrator can register new upstream container ports on the fly with a simple local API call, and active WebSocket streams never experience a single blip in latency.

Choosing Pragmatic Ergonomics Over Legacy Inertia

Swapping NGINX for Caddy across our production fleet taught us how easily engineering teams mistake familiarity for simplicity. NGINX remains a remarkable piece of engineering if you are squeezing every last microsecond out of bare-metal content delivery networks or writing custom Lua traffic filters at massive scale. However, for ninety-five percent of modern containerized architectures, manually wiring up TLS renewals, WebSocket upgrade headers, and multi-tenant virtual hosts is undifferentiated heavy lifting. Moving to Caddy gave us secure defaults, native HTTP/3 performance, and self-healing certificate automation out of the box, letting us spend our engineering hours building product features instead of debugging proxy syntax.