When "no healthy upstream" isn't about the upstream you think

Seeing a no healthy upstream error usually signals a catastrophe. Yet, in a recent incident I investigated, the service would magically recover by the time the user hit refresh.

I was hunting an intermittent glitch in a search backend. We thought we had a "slam dunk" Root Cause Analysis (RCA): a simple explanation paired with a one-line fix. However, this experience highlighted the dangerous gap between a plausible narrative and actual evidence, specifically regarding how a healthy fleet can accidentally sabotage itself.

The Anatomy of the Failure

The architecture was standard: a load balancer (LB) sitting in front of a search backend, where each instance utilized a fixed pool of worker processes.

The Symptoms

• Random Bursts: Requests would fail sporadically with no clear schedule.
• The Error: Users saw a stark no healthy upstream message.
• The Recovery: Everything returned to normal within a few minutes.
• The Latency Profile: p99 latency didn't climb gradually; it hit the LB timeout limit exactly, plateaued, and then crashed back to baseline.

The initial theory was that a heavy periodic background job was pinning the CPU, causing the scheduler to throttle the pod. This would starve request handling, leading the LB to mark the instance as unhealthy and evict it.

Proposed Fix: Increase the CPU limit.

Testing the Hypothesis

I decided to treat the RCA not as a fact, but as a hypothesis. If CPU throttling were the culprit, certain evidence must exist.

Evidence Audit

• Topology Check: The batch indexers (the suspected CPU hogs) ran on entirely different machines. They communicated via the network. A process outside a pod's cgroup cannot trigger that pod's CPU throttling.
• Metric Analysis: Container throttling counters remained flat during the outages.
• Timing Analysis: The batch jobs followed a rigid schedule, but the failures happened at random intervals and far more frequently than the jobs ran.
• Distribution: The issue persisted across various pods and different deployment versions (SHAs).

Theory: CPU Throttling	Observed Evidence	Verdict
Heavy jobs cause throttling	Throttling counters are flat	❌
Jobs trigger the events	Events occur randomly/more often	❌
Pod is resource starved	Indexers are on separate hardware	❌

The theory required too many excuses to remain viable. I abandoned it and returned to the logs.

The "Aha!" Moment in the Logs

Deep in the logs, I found a recurring culprit: ConnectionTimeout: Connection timed out. The stack trace pointed to the datastore client, which was hanging on a socket read.

Crucial Distinction: A CPU-throttled worker is ready to work but is denied CPU cycles. A worker blocked on I/O is off-CPU, waiting for the network while still occupying a slot in the worker pool.

Increasing CPU limits in this scenario is useless; it simply provides more workers to get stuck behind the same slow dependency.

The Mechanics of Saturation

The failure was caused by a lethal combination of two settings:

1. No individual timeouts on datastore calls.
2. Aggressive retries with backoff.

To understand why this kills a service, we look at Little's Law: $L = \lambda W$ Where:

• $L$ = Average number of concurrent requests (concurrency)
• $\lambda$ = Arrival rate of requests
• $W$ = Average time a request spends in the system

Scenario A (Healthy): $\lambda = 200 \text{ req/s}$, $W = 0.04 \text{ s}$ $L = 200 \times 0.04 = 8 \text{ concurrent requests}$ The worker pool handles this easily.

Scenario B (Degraded Datastore): $\lambda = 200 \text{ req/s}$, $W = 20 \text{ s}$ (due to timeouts/retries) $L = 200 \times 20 = 4,000 \text{ concurrent requests}$ The worker pool is instantly exhausted.

The Domino Effect

This leads to head-of-line blocking. Workers stuck on slow datastore calls block fast requests from being processed. Consequently:

1. Latency spikes for all requests, not just the slow ones.
2. The LB sees the timeouts and removes the instance.
3. Traffic shifts to remaining instances, accelerating their saturation.
4. Eventually, no healthy upstreams remain.

The Problem of Wasted Work

The client's total retry time often exceeded the LB's deadline. This meant the worker was still retrying a request that the LB had already abandoned.

# Conceptual Fix: Implement Deadlines
timeout_settings:
  outer_lb_deadline: 30s
  inner_datastore_timeout: 5s # Must be < outer_lb_deadline
  max_retries: 3

The Lesson: Inner operations must complete within the outer request deadline. Ideally, the deadline should be propagated through the entire call chain so every layer knows when a result is no longer useful.

The Self-Recovery Paradox

The system recovered on its own because the failure was metastable. The retries added load to a struggling datastore, creating a feedback loop of timeouts and more retries. Once the LB evicted enough instances, the overall traffic volume dropped sufficiently for the datastore to recover, allowing the remaining pods to clear their queues and the LB to gradually re-introduce the "healthy" instances.