Field notes · Operating & reliability
The CDN gave us an address our own country couldn't reach
A CDN answered DNS with two addresses, one of which was blackholed by Egyptian networks. Browsers picked at random, so roughly every other visit stalled about twenty-one seconds before silently falling back. Every monitor, every curl, and every origin check looked perfectly healthy throughout.
The report was that the site “takes a few seconds before it even starts loading, then finishes instantly” — and only sometimes. On mobile it sometimes never loaded at all. A comparison site on a different domain was fast every time from the same phone, on the same network, which made it look like a problem with the slow site’s code.
Four separate theories were tried and abandoned before the real cause turned up, and the reason is worth more than the fix.
What was actually wrong
The CDN answered A queries with two anycast addresses. From Egyptian ISPs, one of them was completely blackholed on port 443 — the connection attempt was never answered, and no ICMP came back to say so. The other was healthy.
A browser picks one of the two more or less at random. Choose the dead one and you wait out the full TCP retransmit sequence — around twenty-one seconds — before falling back to the working address, at which point the page loads instantly. That is precisely the reported symptom: a long pause, then everything at once, about half the time.
On mobile it was worse. IPv6 was tried first, the IPv6 address shared the bad prefix, and mobile stacks are less patient about falling back. So it read as total failure rather than slowness.
Why every diagnostic said the site was fine
This is the part that cost the time.
- Command-line tools looked healthy. A single connection either wins the coin flip
or retries on a different cadence than a browser does. One
curlproves nothing here. - The uptime monitor was green — and structurally could not have been anything else. It ran on the same infrastructure it was watching, from a region where the route was fine. A monitor cannot see a network path it never takes.
- The origin was blameless. Server, web server, certificates, and the files themselves were uninvolved. The failure happened before the connection was established.
- The answer depended on which resolver you asked. One public resolver handed out the bad pair; another handed out a healthy pair. So “check DNS, then connect” from most places sees nothing at all.
- Real-user data actively misled. The stalls landed in the slowest ten percent as ~21-second values, which look exactly like the measurement artefacts that metric is known to produce. They were dismissed as noise once, on reasonable grounds.
- It correlated with nothing. Every code and configuration change appeared not to work, because the broken layer was IP routing between the visitor and one specific address.
The diagnostic that finds it
From the affected network — and the vantage point is the whole trick, because this is invisible from anywhere else:
- Get the answer set from more than one resolver. They differ, and that difference is a clue rather than an inconsistency.
- TCP-connect to every address in the answer set individually, several times each, measuring connect time.
- One address timing out at ~21 seconds while its siblings connect in ~50ms is the signature. Confirm from another region that the address is alive globally, which proves it is a regional route problem rather than a dead server.
In the browser, the giveaway is that the entire delay sits inside the connect phase. Everything after it is normal.
What changed permanently
The per-address reachability check is now part of the automated site audit: resolve through multiple resolvers, connect to every address returned, and treat any unreachable answer as critical. It runs from the audience’s network, not from the origin.
The monitoring lesson is the one I would keep even if the specific fix became obsolete: a monitor hosted on the thing it monitors cannot detect a regional path failure, by construction. It will report green for a site that half your market cannot open. If availability matters, at least one check has to originate from where your visitors actually are.
And the diagnostic habit: never conclude “measurement artefact” without a live reproduction. The data was correct the first time. It was the interpretation that was comfortable and wrong.
Free to quote with attribution and a link to this page. If you think something here is wrong, tell me — a note that stays wrong is worse than one that was never written.
Working together
Client work starts with a conversation — no commitment, no pitch. If it turns out I'm not the right fit, I'll say so.