A health check that returns 200 while the app is broken
Cloud & DevOps · Aug 2026 · 18 min read
Every target green, every dashboard green, and a third of requests failing. The endpoint was answering honestly — it had simply never been asked anything about the application.
The incident lasted nineteen minutes and the monitoring never noticed. Four ECS tasks behind an ALB, all four reporting healthy for the entire window, while the ingest endpoint returned 502 to about a third of the agents posting to it. We found out because a customer's operations lead told us, which is the worst way to find out anything.
The health check was not broken. It answered every probe correctly and quickly. It just answered a question nobody had wanted asked.
Both series in percent on one axis. The amber line is every dashboard we had; the red line is what the agents were getting. What the probe was actually testing
Some months earlier — reasonably, and with a straight face — somebody had moved the health route into nginx to stop the load balancer's probe from occupying a PHP worker every fifteen seconds.
# Answer the probe at the edge. Costs nothing,
# keeps the app logs clean.
location = /healthz {
access_log off;
return 200 'ok';
}
location / {
fastcgi_pass php:9000;
}A slow third-party call had saturated the FPM pool. All forty children were busy waiting on a socket, the listen backlog filled, and nginx started returning 502 the moment fastcgi_read_timeout elapsed. Meanwhile /healthz never touched FPM at all, so it kept returning 200 in under a millisecond — accurately reporting that nginx was fine, which it was.
The probe and the traffic took different paths, so the probe could only ever report on the path it took. A health check that does not travel the request path proves nothing about the request path.
That is the first rule, and it rules out more than the nginx trick. A check served by a different process, a different thread pool, a different connection pool, or a cached response is measuring something adjacent to your application rather than your application.
Reachability is not capacity: SELECT 1 is the other version of this mistake. It proves the database will accept a connection and answer a trivial question. It says nothing about whether your actual queries return before the client gives up — and "the database is slow" is a far more common outage than "the database is gone". If the check borrows a connection from outside the pool your requests use, it will pass while every real query queues.
So check everything, then
That was my conclusion too, and I have written it down before : health-check the dependencies, not the process. It is good advice that is half of the answer, and the missing half caused a worse incident than the one it fixed.
The check grew a Redis ping. Redis was our cache — not our session store, not our queue broker, just a cache in front of some expensive reads. One afternoon it went away for about ninety seconds during a failover.
Every instance shares the same dependency, so a dependency check fails every instance simultaneously. There is no partial outcome. The band is the dependency outage. Everything to the right of it is the health check's own contribution. Every task failed its check in the same second. The service dropped to zero healthy targets, ECS began replacing all of them, and the replacements came up cold into a stampede with no warm cache and no spare capacity. A ninety-second cache blip became an eleven-minute outage, and the health check did all of it.
The mistake is easy to state once you have made it. I had been treating the check as a *description* of the system's health. It is not. It is an *instruction* to the orchestrator.
A readiness check is a request to be taken out of service. Only name the dependencies you would rather go dark than serve without.
Three questions, three checks
The tension resolves once you stop trying to make one endpoint answer three different questions. Kubernetes names them; ECS with a plain ALB does not, which is why the conflation is so common there.
The distinction that matters is the consequence, not the content: wait, restart, or withdraw. what each one may touch, and why Check May touch Failure means Startup Nothing outside the process Not finished booting — keep waiting Liveness In-process state only Restart this container Readiness Hard dependencies only Stop sending this instance traffic
The rule that does the most work is the liveness one: a liveness check must never touch a network dependency. Restarting your container cannot fix somebody else's database, and if it tries, a dependency outage becomes a fleet-wide restart loop — the same cascade as before, with the added charm that the crash-loop backoff keeps your instances down after the dependency recovers.
On ECS you have to build all three: The ALB target-group probe is a readiness check: failing it withdraws traffic and nothing else. Liveness is the container-level HEALTHCHECK in the task definition, which ECS acts on by replacing the task. Startup is healthCheckGracePeriodSeconds, which suppresses judgement while the app boots. Three different settings, three different files, one concept — and if you only configure the ALB one, you have readiness and nothing else.
Hard and soft is a property of your app, not of the technology
"Is Redis a hard dependency?" has no general answer. As a cache it is soft — you serve slower. As a session store it is hard — you cannot authenticate anyone. Same server, same client library, opposite answers, and the only thing that decides is what your code does when it is missing.
// Hard: we would rather serve nothing than serve this
// wrong. Soft: we degrade and stay in the rotation.
const HARD = [
{ name: 'db', check: () => db.query('SELECT 1') },
];
const SOFT = [
{ name: 'cache', check: () => redis.ping() },
{ name: 'search', check: () => os.ping() },
];
app.get('/readyz', async (req, res) => {
const hard = await settleAll(HARD, 800);
const soft = await settleAll(SOFT, 800);
// Soft results are reported, never gated on.
const ok = hard.every((r) => r.ok);
res.status(ok ? 200 : 503).json({ hard, soft });
});Reporting the soft results in the body rather than the status code turns out to be the most useful line in the file. The orchestrator reads only the status; a human reading the body during an incident gets the whole dependency picture from one curl, without needing the check to have an opinion about it.
Never let a check be transitive
If service A's readiness calls B, and B's readiness calls C, then C having a bad minute withdraws A from its load balancer. You have coupled the availability of unrelated services through their health endpoints, which is the exact opposite of what a service boundary is for.
A health check answers for the instance serving it. It never answers for anything downstream of that instance. If A genuinely cannot function without B, that belongs in A's own hard-dependency list as a direct connectivity check with a timeout — not as a call to B's health endpoint, which additionally pulls in everything B depends on.
Probes multiply: The probe interval is per target, not per service. Twenty tasks at fifteen seconds is eighty checks a minute; if each fans out to three dependencies you have added two hundred and forty extra queries a minute to systems that were not consulted about it. Deep checks that are individually cheap are collectively a load test you run against yourself forever.
The timeout that has to be shorter than the other timeout
A check with no timeout is worse than no check. When the dependency it queries hangs, the check hangs with it, the probe times out at the load balancer instead, and you get the correct answer for the wrong reason — slowly, and identically for every instance.
800ms per dependency, 2s for the handler, 5s probe timeout, 15s interval. Each one ends inside the next. Each layer must finish comfortably inside the one above it. If the per-dependency timeout is not strictly less than the probe timeout, the probe cannot distinguish "the dependency is slow" from "this instance is wedged", and those two have opposite correct responses.
Two more numbers worth setting deliberately: an unhealthy threshold above one, so a single blip does not withdraw an instance, and a deregistration delay long enough to finish in-flight requests but short enough that a rolling deploy does not crawl. We use three and thirty.
The tier with no health check at all
Everything above is about things that answer HTTP. Our queue consumers do not, and for a long time nothing watched them, which meant the web tier could be flawlessly green while no interval had been processed in forty minutes.
A worker cannot be probed, so it has to report. The pattern that has held up is a heartbeat the worker writes after each successful batch, and a check that reads it.
// In the consumer, after a batch is acknowledged:
await redis.set(`hb:worker:${id}`, Date.now(),
'EX', 300);
// Alert on the heartbeat AND on queue depth. Either
// alone lies: a wedged consumer can hold a stale
// heartbeat, and an empty queue can mean nothing is
// being produced.
stale = now - heartbeat > 120_000;
piling = queueDepth > 5_000 && depthRising;Here the check is inverted: the alert is the product, not the endpoint. Nothing restarts the worker automatically, because a consumer that is merely behind should be left alone to catch up, and one that is genuinely wedged needs a person to look at why before it is cycled.
What I would do again
Make the check share the request path. Everything else in this post is a refinement of that one property, and it is the one that was violated in the incident that cost us the most.
The correction I would make to my own earlier advice is smaller than it sounds but changes the outcome completely. "Check your dependencies" is right; "check every dependency you have" is how you build a system that takes itself down. Write the hard list deliberately, keep it short, and treat every name on it as a statement that you would rather serve nothing than serve without it. Most of the time that list has exactly one entry.
References Takeaways
A check that does not travel the request path proves nothing about it — no edge-served static 200, no separate connection pool, no cached result. SELECT 1 proves reachability, not capacity. "Slow" is a more common outage than "gone". A readiness check is an instruction to withdraw, not a description of health. Every instance shares its dependencies, so a dependency check fails the whole fleet at once. Liveness must never touch a network dependency — restarting your container cannot fix someone else's database. Hard versus soft is a property of your code's behaviour without the dependency, not of the technology. Gate on hard only; report soft in the body. Never call another service's health endpoint from your own; check your direct connectivity to it instead. Every timeout must be strictly shorter than the one above it, or you cannot tell a slow dependency from a wedged instance. Workers cannot be probed, so have them write a heartbeat, and alert on heartbeat age together with queue depth. All notes · Shehzad Aslam
