A 500 That Survived Ten Successful Deploys

A specific page returns 500. Other pages load. The site builds clean locally. All referenced images exist. Every internal link resolves. CI is green. It turns out the bug was introduced ten releases ago.

Is / Is Not

How I troubleshoot these days has its roots in my support engineer days at Sun Microsystems, where we used the Kepner-Tregoe methodology from The Rational Manager.

The core tool for problem analysis is an "Is / Is Not" matrix. Before you guess at causes, you write down what the problem IS and what it IS NOT, and then you look for what changed at the boundary.

Check out Is - Is Not analysis needs that Is Not side too! for a good walkthrough.

Applied here (I did not do it formally, but I had this in mind as I was troubleshooting):

The "Is" column says: one URL, one deploy, Cloudflare-side. The "Is Not" column rules out the codebase, the build step, and a general CDN failure.

The search

I checked the article file. Valid TOML frontmatter, clean markdown, no characters that would break the Zola renderer. Ran zola build:

Building site...
-> Creating 25 pages (0 orphan) and 11 sections
Done in 350ms.

Also zola check ran fine, it only listed some broken links, but that was not part of the problem.

So no errors. Every referenced image existed. Every internal link resolved. The local build told me nothing was wrong.

The Cloudflare Pages dashboard did. The current deploy showed green. But scrolling back through deploy history: one deploy about ten releases ago had failed. It was logged as a failure. I saw it at the time, watched the next push succeed, and moved on. That was the mistake.

The commits visible in that history: "Chore: Trigger Cloudflare Pages deploy" and "test build without last article". These were troubleshooting attempts. More pushes followed, each green, but failing to clear the corrupted edge routing state created by the failed build.

Forcing a full Cloudflare Pages deploy

First, I tried purging the Cloudflare cache from the dashboard. The 500 persisted. The issue wasn't the cache layer; the edge was serving a stale, corrupted build state left behind by that failed deployment ten releases ago.

I asked ChatGPT for help. It suggested running npx wrangler pages dev ./dist. While that is just a local development command, it put Wrangler, Cloudflare's official developer CLI, on my radar.

The connection clicked. Because the standard Git-triggered pipeline was repeatedly passing over the existing edge state, I needed to force an update. wrangler pages deploy bypasses the automated CI pipeline entirely, pushing every file completely fresh to the edge.

Fix:

zola build && wrangler pages deploy public --project-name belderbos-dev

Result: the 500 disappeared, style.css loaded, and the page rendered correctly.

AI was really useful here, it nudged me towards using the right tool. It narrowed the search space, however deeper environment understanding and knowledge of the ecosystem was required to understand why that tool was the right one.

So by all means use AI to help troubleshoot, but also use tools like Is / Is Not analysis to structure your thinking, and understand the environment you are working in so that you can evaluate the hints AI gives you.

And as always, I am a big proponent of doubling down on fundamental understanding, in this case of cloud deployments. This does not go away with AI, it's actually a pre-requisite for getting more mileage out of AI tools.

The real lesson

From the Zen of Python:

Errors should never pass silently. Unless explicitly silenced.

The same principle applies to your deployment pipeline. A failed deploy is not a closed incident just because the next one succeeded. The broken state it wrote to the edge is still there. Treat deploy failures as open incidents until you have verified the output on the edge, not just waited for the next green run.

Highly recommended to set email alerts on Cloudflare for deploy failures (I had this set but somehow I didn't get notified). I aso documented this 'override' in my site's Makefile:

deploy:
    zola build && wrangler pages deploy public --project-name belderbos-dev