Console

—

—

Speed

—
no replies yet

Tokens generated

—

Anomalies

—

Read from disk

—

Where the memory goes

——
experts in memory —
rest of the model —
room for one step —
this conversation —
unused —

—

Model

——
— unmodified
tokens/s
—
requests
—
ttft
—

Cache configuration synced

Changing these reloads the model. Nothing else is affected.

Experts kept in memory ——
More resident means fewer reads from disk and more speed, and more memory. The rest are streamed.
1128
Longest reply ——
How much room to leave for one answer. Every token of a reply costs KV cache.
256—
Memory for remembered prompts ——
Set aside so a follow-up question does not read the whole conversation again. It comes out of the same budget as the experts.
off—

Remembered prompts

The model keeps what it has already read, so a follow-up question does not read the whole conversation again. Measured on a document plus two questions: 11.2s to the first word, then 1.1s and 0.8s.

Tokens not re-read
—
nothing reused yet
Reused
—
Memory held
—

Activity

What the engine has been doing. It reconfigures itself while it runs -- handing memory back, forgetting prompts, rebuilding the pool -- and each of those changes how fast the next reply arrives.

Live
waiting for the engine to do something

This machine