SOURCE C: Health Checks, Graceful Degradation, and Load Shedding

Reliable microservices must expose accurate health signals. There are three standard check types: liveness (is the process alive and not deadlocked?), readiness (is the service ready to accept traffic?), and startup (has initialisation completed?). Kubernetes distinguishes all three; mixing them up — for example using a readiness probe that hits the database, causing the pod to be removed from rotation during a slow DB query — is a common source of accidental outages.

Graceful degradation means returning a useful partial response rather than a hard error when dependencies fail. A product page that cannot reach the recommendation service should still render the product; it simply omits the "Customers also bought" section. Implementing this requires clear API contracts that distinguish mandatory from optional data, and feature flags or fallback logic at the caller. Cache-served stale data is another degradation strategy: return the last known good response with a staleness header rather than an empty response.

Load shedding is the controlled rejection of excess traffic to protect service health. When a service detects that its queue depth or latency is climbing beyond a safe threshold, it should start returning 503 responses with a Retry-After header rather than accepting all traffic and degrading for every user. Priority-based shedding — where premium traffic is preserved and background traffic is dropped first — is the advanced form and is used by services like Google's Bigtable. Adaptive concurrency limits (used in Envoy and Netflix's concurrency limiter) automate this by continuously measuring latency and adjusting the in-flight request limit dynamically.

Testing these mechanisms requires fault injection in production or staging. Chaos engineering tools (Chaos Monkey, Gremlin, Chaos Mesh) deliberately kill instances, inject latency, and partition networks to verify that health checks, circuit breakers, and degradation paths actually fire. Without adversarial testing, reliability mechanisms frequently turn out to be misconfigured or untriggered paths.
