casty · performance sweep · replicated virtual actors
39 load points sweeping cluster size, concurrency and actor count, at peak capacity (no rate cap), 30 s each. Against replicas=3, write=MAJORITY. Throughput is bounded by quorum replication latency, not by CPU: more actors buy parallelism, more nodes charge latency, and a single hot key is a hard ceiling at ~5k ops/s.
Each line is a cluster size. On the left, 50 writers; on the right, 250. 1 actor is the hot-key floor — a single owner serializes everything. Throughput jumps as soon as the keyspace spreads out, saturates at a plateau, and decays ~5–7% at 20 000 actors (activation/memory cost). Concurrency from 50→250 writers lifts the plateau ~40% on the small cluster; on the larger ones the gain vanishes — it becomes latency-bound.
ops/s vs actors (x log)
ops/s vs actors (x log)
With 50 writers (left) the tail is placid: p99 stays at 6–9 ms across the whole range. With 250 writers (right) and few actors, concurrency piles up in the FIFO mailboxes and the tail explodes: p99 of 120 ms (15 nodes) and 218 ms (50 nodes) at 25 actors. From ~100 actors on, the tail collapses to ~23–30 ms. Rule of thumb: actors ≳ 4× writers to keep the tail flat.
p99 ms vs actors (y log)
p99 ms vs actors (y log)
Plateau throughput (100 actors) against node count. With replicas=3 fixed, adding nodes does not add replication parallelism for a given keyspace — it only spreads owners and backups across more distant hosts, stretching the quorum round-trip. More nodes = more latency = less throughput per client.
vs cluster size
1 actor delivers ~5.1k ops/s regardless of nodes (5.3k / 5.2k / 5.1k) or of writers (250 writers give the same ~5k, only with p99 going 50→85 ms). The single owner serializes handlers (FIFO mailbox, 1 at a time) and every write pays the MAJORITY round-trip. Neither concurrency nor a larger cluster breaks through this — only sharding.
Plateau: 15 nodes → 25k, 50 nodes → ~15k, 100 nodes → ~13k (−48% from 15 to 100). The bottleneck is quorum replication latency, not node CPU: spreading owners+backups across more hosts stretches the write=MAJORITY path. A large cluster is about state capacity (more actors fit), not about ops/s per client.
Well sharded, 50→250 writers raises the plateau ~40% on the small cluster (18k→25k). Throughput is ≈ concurrency ÷ latency: from ~25 actors on in the 50-writer case, the ceiling becomes the number of writers in flight itself. Actors beyond what is needed do not help — they just must not be too few.
250 writers on 25 actors: median still healthy (~5–7 ms) but p99 of 120–218 ms — deep queues in the few mailboxes. The median hides it; the p99 exposes it. Size the keyspace by concurrency, not by average load.
Jumping from 2.5k to 20k actors costs ~5–7% of throughput and a few ms of p99 (on-demand activation + memory for more activations). No cliff, no errors. Virtual actor density holds up in the tens of thousands.
Zero write errors across 18.7 million acked adds over the entire grid — every ack was reflected in the final value. Pressure showed up as latency, never as loss.
ops/s (p99 in ms). Cell in red = hot-key floor; cyan = peak.