FaultRay Chaos Simulation Report

Generated 2026-04-02 13:29 UTC

Executive Summary

76 Resilience
6
Components
1
Critical
27
Warnings
106
Passed
Score Explanation

The resilience score measures structural health: single points of failure (SPOFs), resource utilization headroom, and dependency chain depth.

1 critical scenario(s) detected cascade failures that could cause widespread outages.

Critical Findings

Traffic spike (10x) 10.0/10
Traffic DDoS-level (10x normal) across all entry points
Cascade path
DOWN nginx (LB) Capacity exceeded: 250% (max 100%)
DOWN api-server-1 Capacity exceeded: 250% (max 100%)
DOWN api-server-2 Capacity exceeded: 240% (max 100%)
DOWN PostgreSQL (primary) Capacity exceeded: 260% (max 100%)
DOWN Redis (cache) Capacity exceeded: 220% (max 100%)
DOWN RabbitMQ Capacity exceeded: 200% (max 100%)

Warnings

Traffic spike (5x) 6.4/10
Traffic viral event (5x normal) across all entry points
Cascade path
DOWN nginx (LB) Capacity exceeded: 125% (max 100%)
DOWN api-server-1 Capacity exceeded: 125% (max 100%)
DOWN api-server-2 Capacity exceeded: 120% (max 100%)
DOWN PostgreSQL (primary) Capacity exceeded: 130% (max 100%)
DOWN Redis (cache) Capacity exceeded: 110% (max 100%)
OVER RabbitMQ Near capacity: 100%
Cache stampede: redis + 5.0x traffic 5.4/10
Cache redis fails, all requests hit DB. Traffic at 5.0x. Thundering herd pattern.
Cascade path
DOWN nginx (LB) Capacity exceeded: 125% (max 100%)
DOWN api-server-1 Capacity exceeded: 125% (max 100%)
DOWN api-server-2 Capacity exceeded: 120% (max 100%)
DOWN PostgreSQL (primary) Capacity exceeded: 130% (max 100%)
DOWN Redis (cache) Capacity exceeded: 110% (max 100%)
OVER RabbitMQ Near capacity: 100%
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +10s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: nginx + app-1 + postgres 5.0/10
Simultaneous failure of nginx + app-1 + postgres
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
Triple failure: nginx + app-1 + redis 5.0/10
Simultaneous failure of nginx + app-1 + redis
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: nginx + app-1 + rabbitmq 5.0/10
Simultaneous failure of nginx + app-1 + rabbitmq
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: nginx + app-2 + postgres 5.0/10
Simultaneous failure of nginx + app-2 + postgres
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +5s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
Triple failure: nginx + app-2 + redis 5.0/10
Simultaneous failure of nginx + app-2 + redis
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +5s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: nginx + app-2 + rabbitmq 5.0/10
Simultaneous failure of nginx + app-2 + rabbitmq
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +5s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: nginx + app-1 + app-2 4.7/10
Simultaneous failure of nginx + app-1 + app-2
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +0s
Triple failure: nginx + postgres + redis 4.6/10
Simultaneous failure of nginx + postgres + redis
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +5s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: nginx + postgres + rabbitmq 4.6/10
Simultaneous failure of nginx + postgres + rabbitmq
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +5s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: nginx + redis + rabbitmq 4.6/10
Simultaneous failure of nginx + redis + rabbitmq
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +10s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +50s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: app-1 + app-2 + postgres 4.6/10
Simultaneous failure of app-1 + app-2 + postgres
Cascade path
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +0s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
Triple failure: app-1 + app-2 + redis 4.6/10
Simultaneous failure of app-1 + app-2 + redis
Cascade path
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: app-1 + app-2 + rabbitmq 4.6/10
Simultaneous failure of app-1 + app-2 + rabbitmq
Cascade path
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Total infrastructure meltdown 4.6/10
Complete failure of all components simultaneously (worst case, extremely unlikely — simultaneous all-down)
Cascade path
DOWN nginx (LB) Component failure (simulated)
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +0s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +50s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: app-1 + postgres + redis 4.2/10
Simultaneous failure of app-1 + postgres + redis
Cascade path
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: app-1 + postgres + rabbitmq 4.2/10
Simultaneous failure of app-1 + postgres + rabbitmq
Cascade path
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: app-1 + redis + rabbitmq 4.2/10
Simultaneous failure of app-1 + redis + rabbitmq
Cascade path
DOWN api-server-1 Component failure (simulated)
WARN nginx (LB) Dependency api-server-1 is down, remaining replicas handling load (1 left) +5s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +50s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: app-2 + postgres + redis 4.2/10
Simultaneous failure of app-2 + postgres + redis
Cascade path
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +5s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Triple failure: app-2 + postgres + rabbitmq 4.2/10
Simultaneous failure of app-2 + postgres + rabbitmq
Cascade path
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +5s
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: app-2 + redis + rabbitmq 4.2/10
Simultaneous failure of app-2 + redis + rabbitmq
Cascade path
DOWN api-server-2 Component failure (simulated)
WARN nginx (LB) Dependency api-server-2 is down, remaining replicas handling load (1 left) +5s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +50s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Pair failure: postgres + redis 4.0/10
Simultaneous failure of postgres and redis
Cascade path
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +5s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
Pair failure: postgres + rabbitmq 4.0/10
Simultaneous failure of postgres and rabbitmq
Cascade path
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +5s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +55s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Pair failure: redis + rabbitmq 4.0/10
Simultaneous failure of redis and rabbitmq
Cascade path
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +10s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +50s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Triple failure: postgres + redis + rabbitmq 4.0/10
Simultaneous failure of postgres + redis + rabbitmq
Cascade path
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +5s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s
DOWN RabbitMQ Component failure (simulated)
WARN api-server-1 Async dependency RabbitMQ is down, queue building up +50s
WARN api-server-2 Async dependency RabbitMQ is down, queue building up +0s
Cascading meltdown (root-cause) 4.0/10
Critical root-cause components fail (postgres, redis), cascade engine determines realistic spread through dependency graph
Cascade path
DOWN PostgreSQL (primary) Component failure (simulated)
WARN api-server-1 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +5s
WARN api-server-2 Dependency PostgreSQL (primary) is down, remaining replicas handling load (2 left) +0s
DOWN Redis (cache) Component failure (simulated)
WARN api-server-1 Optional dependency Redis (cache) is down +5s
WARN api-server-2 Optional dependency Redis (cache) is down +0s

Infrastructure Overview

Name Type Host Replicas Utilization CPU Memory Disk Connections
nginx (LB) load_balancer web01:443 2 25% 15% 20% 25% 0
api-server-1 app_server app01:8080 3 25% 22% 25% 22% 200
api-server-2 app_server app02:8080 3 24% 20% 24% 20% 180
PostgreSQL (primary) database db01:5432 2 26% 20% 26% 0% 0
Redis (cache) cache cache01:6379 2 22% 8% 22% 0% 100
RabbitMQ queue mq01:5672 2 20% 10% 20% 15% 25

Dependency Map

nginx (LB) api-server-1 api-server-2 PostgreSQL (primary) Redis (cache) RabbitMQ
FaultRay — Virtual Infrastructure Chaos Engineering Simulator