When we get it wrong

The API went down for a morning and our monitor didn't notice.

7 September 2026

What happened: For about ten hours our API returned server errors — agents pulling their scheduled probes got a Cloudflare 'over resource limits' response instead of their tasks, so no verification ran in that window. The public site stayed up, which is why nothing looked wrong from the outside.
Root cause: Two failures stacked. First, the per-request work on the probe path had grown — extra database round-trips and large modules loaded on the hot path — and a burst of cold starts right after a deploy tipped the worker over its CPU limit. Second, and worse: our uptime monitor only checked the static site, which is served off a separate path and stayed up the whole time, so it never saw the API failing underneath.
Fix: We added a real, CPU-cheap health endpoint that fails exactly when the worker is over its limit, and pointed the monitor at it — an API error now reaches us within fifteen minutes instead of never. The heavier structural fix to the probe path (serve the large static file off the CPU path, precompute the coverage draw) is scoped separately.
Lesson: A monitor that watches the part that can't fail isn't a monitor. We now canary the exact surface that broke — the API worker — not the static shell in front of it.