"It returned 200" is not the same as "it deployed"

The deploy was the standard atomic swap: move the live directory aside, move the new build into its place. Nothing failed. No downtime, no errors in any log. Verification requested all seven new URLs and got 200 on every one.

All seven were fictional. The server had been serving the previous build the entire time, and carried on doing so.

Why the swap did nothing

The web server ran in a container that bind-mounted the site’s specific directory. A bind mount to a named directory pins that directory’s inode when the container starts.

Renaming it does not move the mount. It renames the thing the container is still reading from, and creates a brand-new directory at the old path that the container cannot see. From the host it looks like the deploy succeeded — the files are exactly where you expect. From inside the container, nothing changed and nothing ever will, until it restarts.

Why verification confirmed it anyway

This is the part that should worry anyone who verifies deploys with status codes.

The site had a catch-all rule — the ordinary single-page-app fallback that serves index.html for anything unmatched. So a request for a page that exists in neither build still returns 200, with Content-Type: text/html, and a body. Every check passed. The check was incapable of failing.

Status codes verify that a server responded. They say nothing about what it responded with, and on any site with a catch-all they cannot distinguish a deployed page from a missing one.

The rules that came out of it

Match the deploy shape to the mount shape. If the container mounts a specific directory, sync into that directory and never rename it. Directory swaps are only safe when the mount is the parent. Check the mount before choosing the deploy method — it takes a moment and this failure mode is silent.

Verify content, not status. Post-deploy checks now assert something that exists only in the new build: a marker string, a changed asset with its Content-Type and length, a modified timestamp. A 200 on a catch-all site is not evidence.

Beware the backup that isn’t one. The “backup” the failed swap produced was the live directory itself, renamed — the same inode the container was still serving from. It was not a copy of anything. A real backup is cp -a, taken before the deploy, and confirmed to be a separate set of bytes.

The general version

This belongs to a family: changes that appear verified because the verification was structurally unable to detect the failure. Its close cousin is a cache-busted asset whose version parameter never got incremented — the file on disk is correct, the check reads the file, and visitors keep receiving the old one from cache for weeks.

Both share a shape worth recognising: the check and the failure are on different sides of the thing being tested. When something reports success, the useful question is not “did it pass?” but “what would this check have done if it had failed?” If there is no answer, the check is decoration.