Metadata-Version: 2.5
Name: sous-mcp
Version: 0.6.0
Summary: The sous-chef for Claude's kitchen: an MCP daemon that delegates mechanical, volume-heavy coding tasks to a sandboxed local MLX worker on Apple silicon, so heavy Claude Code use stretches further on the same plan
Project-URL: Repository, https://github.com/krcm0209/sous
Project-URL: Issues, https://github.com/krcm0209/sous/issues
Author-email: krcm0209 <17315774+krcm0209@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
License-File: LICENSES/Apache-2.0.txt
License-File: THIRD_PARTY_NOTICES.md
Keywords: ai-agents,apple-silicon,claude,claude-code,delegation,local-llm,mcp,mcp-server,mlx,model-context-protocol,qwen
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: MacOS X
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.14
Requires-Dist: anyio>=4
Requires-Dist: httpx>=0.28
Requires-Dist: huggingface-hub
Requires-Dist: mcp<3,>=2.0
Requires-Dist: mlx-lm>=0.31
Requires-Dist: mlx-vlm>=0.6.16
Requires-Dist: mlx>=0.31
Requires-Dist: psutil>=7
Requires-Dist: rich>=15
Requires-Dist: sse-starlette>=3.4
Requires-Dist: starlette>=1.6
Requires-Dist: textual<9,>=8.2
Requires-Dist: tomlkit>=0.13
Requires-Dist: uvicorn>=0.31
Description-Content-Type: text/markdown

# sous

<!-- mcp-name: io.github.krcm0209/sous -->

The sous-chef for Claude's kitchen: delegate mechanical, volume-heavy coding
tasks from Claude Code / Claude Desktop to a local MLX model on your Mac.
Claude designs the menu; sous does the prep — in a sandboxed, auditable,
autonomous tool loop. The sandbox is application-level, not an OS jail
([what that means](#security-model)). You (and Claude) review everything it
cooks.

## Why

Heavy Claude Code use runs into plan usage limits, and most of what consumes
them is generated output. sous exists to stretch that budget: the mechanical,
volume-heavy output comes from a local model at zero marginal cost, while
Claude spends its much cheaper input-side attention writing instructions and
reviewing the resulting diff. Claude stays the head chef, and the same plan
carries further into the week.

This is a hybrid local + cloud arrangement, deliberately not an
all-or-nothing switch to a local model: pointing Claude Code itself at a
local endpoint trades away the frontier reasoning you are paying for, while
sous offloads only the small, mechanical work your Apple silicon Mac can
handle on its own.

## How sous compares

Other ways to put local models next to Claude Code make a different trade:

- **Full-local replacements** point Claude Code (or a fork of it) at a local
  endpoint — every task drops to local-model quality, including the ones you
  wanted a frontier model for.
- **Routers/proxies** swap models per request, but the work still runs
  synchronously inside your session, and the only thing between the local
  model and your files is a permission ruleset calibrated for a frontier
  model's judgement.
- **Subagent/skill delegates** hand tasks to a local model with your
  session's permissions and none of the rest: no persistent queue, no
  project-root confinement, no allowlist scoped to a model you trust less,
  no audit trail.

sous is the hybrid: Claude keeps the reasoning, and an asynchronous queue
hands the mechanical work to a local worker that is sandboxed, budgeted,
approval-gated, and journaled — with the diff always reviewed before it
counts.

The cost is capability, and it is deliberate. The worker runs its own loop
over eight fixed tools (`read_file`, `write_file`, `edit_file`, `list_dir`,
`glob`, `grep`, `run_command`, `finish`); it cannot reach your MCP servers,
your skills, or your hooks, all of which a subagent delegate inherits. sous
buys confinement, persistence, and a reviewable diff by giving the local
model a much smaller world to work in. Work that genuinely needs the harness
is work to keep in Claude.

## Requirements

- Apple silicon Mac (tested on M-series with 64 GB; the default model needs
  ~18 GB free unified memory — for 16 GB machines, see
  [Smaller machines](#smaller-machines))
- Python 3.14 (standard build) via [uv](https://docs.astral.sh/uv/)
- Claude Code or Claude Desktop with a Pro/Max plan

## Install

```bash
uv tool install sous-mcp   # PyPI package name; the CLI it installs is `sous`
sous install-launchd       # start at login, keep alive (recommended)
claude mcp add --transport http sous http://127.0.0.1:8383/mcp
```

From a checkout, `uv tool install .` works instead of the PyPI package.
(The bare `sous` name on PyPI is an unrelated, abandoned placeholder — the
package you want is `sous-mcp`.)

### Claude Desktop

`claude_desktop_config.json` launches MCP servers as stdio subprocesses, so it
cannot take the HTTP URL above. Use `sous mcp`, which bridges stdio to the
daemon:

```json
{
  "mcpServers": {
    "sous": { "command": "sous", "args": ["mcp"] }
  }
}
```

`sous mcp` holds no state and loads no model — it forwards messages to the one
daemon. Open several clients and they share it, so the model is resident once
no matter how many are connected. If no daemon is running it starts one, which
is what makes this work without `sous install-launchd`.

Discovery needs no extra setup: the daemon publishes MCP server instructions
that clients put in front of Claude, saying when to delegate and why.
Optionally install the delegation skill as a supplement — a fuller playbook
(mirroring delegations into Claude's task list, approval etiquette,
restarting a downed daemon):

```bash
cp -r skills/delegating-to-local ~/.claude/skills/
```

First delegation downloads the model (~16.1 GB for the default) — one time.

### Managing the daemon

```bash
sous status             # is it up, and what has it been doing
sous top                # watch it live (alias: sous status --watch); q quits
sous statusline         # one line for Claude Code's status bar (below)
sous wait <task-id>     # block until a task finishes or needs approval
sous claude             # Claude Code with local subagents (gateway mode, below)
sous stop               # stop it (see below)
sous uninstall-launchd  # stop it starting at login, and remove the agent
```

`sous stop` deliberately refuses when launchd is managing the daemon, because
`KeepAlive` would restart it a second later and the command would look like it
did nothing. It tells you which command you actually want. Stopping is for
daemons nothing is supervising — including one `sous mcp` started for you.

Stopping the daemon also ends any running `sous mcp` bridges; their clients
reconnect and start a fresh one on the next call.

The daemon writes one log, `~/.sous/daemon.log`, both streams. Every line
sous emits reads `2026-09-10T19:26:14.025Z INFO sous.gateway: …` — UTC
timestamp with milliseconds, then `INFO` (served or forwarded), `WARNING`
(a request sous refused: a 4xx, a 529, a client gone while queued, tools it
had to drop) or `ERROR` (a failure sous produced), then which part spoke.
Library lines (`mcp.…`, `uvicorn.error`, `huggingface_hub…`) take the same
shape, and `warnings.warn` arrives as `WARNING py.warnings`.
Earlier releases split stdout and stderr into `daemon.log` and
`daemon.err.log`; re-run `sous install-launchd` once — it boots the old job
out, waits up to 30 s for its daemon to free the lock, folds `daemon.err.log`
onto `daemon.log` (or only removes the old name when it was already a link to
`daemon.log`), removes it, writes the new plist and bootstraps it. That re-run
needs the daemon stopped first if it was started by hand (`sous serve`, the
normal case on a machine with no launchd job installed yet): finding the lock
still held, the command refuses and writes nothing — `sous stop`, then retry.
Once the old job is unloaded, a failure before the new one loads (a daemon
still exiting after the wait, a fold or plist write that fails) says the job
is unloaded and exits nonzero; fix the cause and re-run. A failed
`launchctl bootstrap` after the plist is written prints the bootstrap command
to run by hand and exits nonzero rather than claiming success.

## What Claude gets

| Tool | Purpose |
|---|---|
| `delegate_to_local_model` | queue a self-contained task (returns immediately) |
| `task_status` | poll progress / queue position / approval requests |
| `task_result` | fetch report, changed files, verify output, diff |
| `cancel_task` | stop a queued or running task |
| `respond_to_command_request` | approve/deny a non-allowlisted command |
| `server_status` | engine, live turns, queue and config — the status document, minus the recent lists GET /sous/status adds |

## Gateway mode (experimental)

sous can also stand between Claude Code and `api.anthropic.com`: with
`[gateway].enabled = true` the daemon serves Anthropic's Messages API on the
same `127.0.0.1:8383`, answers requests for the model id `sous-local` with the
local model, and forwards everything else — the main loop's requests, the
startup probe, usage and telemetry calls — to the real API untouched. That is
the hybrid in issue #41: a frontier main loop on your subscription, Task-tool
subagents on the local model, one `ANTHROPIC_BASE_URL`, and a frontier model
always reviewing the local model's work.

```bash
sous claude                            # Claude Code, subagents served locally
sous claude -p "summarize README.md"   # every argument passes through to claude
```

`sous claude` asks the running daemon for its effective gateway settings —
the config file may have been edited since it started, and only a restart
applies it — and refuses if no daemon answers, if the gateway is off, or if
the daemon predates routing. It says so when the file and the daemon
disagree, and uses the daemon's values. Then it replaces itself with
`claude`, having set:

| Variable | Value | Why |
|---|---|---|
| `ANTHROPIC_BASE_URL` | `http://127.0.0.1:8383` | one endpoint; the gateway routes on the requested model id |
| `CLAUDE_CODE_SUBAGENT_MODEL` | `sous-local` (the first `local_models` entry the daemon reports) | the default model for Task-tool subagents; the main loop keeps its `claude-*` id |
| `CLAUDE_CODE_SUBAGENT_MODEL_FORCE` | `1` | the override: since Claude Code 2.1.26x a built-in agent's own `model:` (Explore, for one) or a per-spawn model beats the default above; this applies the default to every subagent regardless |
| `CLAUDE_CODE_MAX_CONTEXT_TOKENS` | the daemon's `max_context_tokens` | Claude Code has no built-in size for `sous-local`; it honours this variable only for non-`claude-*` ids, so the main loop is unaffected |
| `API_TIMEOUT_MS` | `3000000` | a cold model load plus a long prefill takes minutes |

plus `--disallowedTools LSP` unless you pass your own `--disallowedTools` (a
language server connecting mid-session appends its schema to every request
and re-prefills the conversation). It sets **no** `ANTHROPIC_AUTH_TOKEN`,
`ANTHROPIC_API_KEY` or `ANTHROPIC_DEFAULT_*_MODEL`: either credential
variable switches Claude Code from your subscription login to API-credit
billing, and the tier variables would pull the main loop onto the local
model. If your shell already exports a credential variable, `sous claude`
warns and launches anyway — that is your billing decision, not sous's (the
same for an inherited `ANTHROPIC_DEFAULT_*_MODEL`, which would pull that tier
off the upstream). It
also leaves `CLAUDE_CODE_AUTO_COMPACT_WINDOW` alone: that setting is global,
and pinning it to the local window would make the frontier main loop compact
far too early; the subagent's window is bounded by
`CLAUDE_CODE_MAX_CONTEXT_TOKENS` instead.

Before it execs, `sous claude` asks the daemon to keep the model loaded for
the session: it `POST`s `/sous/hold` with its own process id and start time,
and since `exec` keeps the process id, the holder *is* the Claude Code
process. If nothing is loaded the daemon starts loading right away, on its
own thread — the launcher prints `sous claude: preloading <model>; held
while this session runs` (or `already loaded`) and does not wait, so the
first subagent turn finds the weights resident or the load already under
way instead of paying it. The daemon's idle sweep checks its holders on
every poll between delegated tasks: when the `claude` process exits, the
hold goes with it at the next sweep — within a second on an idle daemon; a
delegated task in flight defers the sweep until it finishes — and the idle
clock restarts from that moment, so quitting and relaunching inside
`[model].idle_unload_minutes` never reloads either. A hold the daemon
refuses is a warning, not a stop. Two caveats: a daemon restart forgets its
holders (the model then unloads after `idle_unload_minutes` and the next
subagent turn reloads it); and `sous claude --help`, `-h`, `--version` and
`-v` skip the hold, while any other invocation that exits at once (`sous
claude mcp list`, say) holds like a session — the load it may start is one
nobody waits for, and the weights then stay resident for a fresh
`idle_unload_minutes`.

The preflight itself is plain HTTP. The daemon's own routes live under
`/sous/` on the same port, loopback-only like the gateway's, whether or not
the gateway is on:

- `GET /sous/status` — one JSON document: `engine` (`loaded`, `loading`,
  `model_id`, `idle_seconds`, `holders`, `memory_gb`, the `prompt_cache`
  counters), `inflight` (the turn the model is serving right now — its
  `msg_` id, phase, tokens so far, rate and ETA — usually empty or one
  entry, ordered with the turn on the pass first and then the queue in
  arrival order), `queue` (delegated task counts), `recent_turns` (the last 50,
  every field of the turn line below), `recent_tasks` (the last 10) and
  `config`. The MCP `server_status` tool returns the same document without
  the two `recent_*` lists.
- `GET /sous/events` — the same document as a Server-Sent Events stream:
  once at connect, then whenever the turn in flight changes (at most ten
  times a second) and at least once a second, with a `ping` every 10 s.
- `POST /sous/hold` — what `sous claude` posts (above).

A `404` from `/sous/status` means the running daemon predates this CLI (the
route did not exist); `sous claude` says so and exits 1 — restart the daemon
from the same install (`sous stop`, then `sous serve` or `sous
install-launchd`). There is no compatibility mode: the CLI and the daemon
ship as one package.

**`sous top`** (or `sous status --watch`) is that stream in a terminal —
open it beside a `sous claude` session and watch a subagent's order go from
the rail to the plate — a perforated slip with a timer dial, a pixel chef
whose face is the state, the walk-in's counters, and the recent orders:

![sous top at 100×30: the order slip mid-decode with a second order queued, THE LINE with the chef, and the recent orders](tests/snapshots/sous-top-100x30.svg)

*`sous top` at 100×30, mid-decode with a second order queued — the render the test suite pins, so the picture is always the current one.*

`q`, `Esc` or `Ctrl-C` leave the screen exactly as it was; `enter` opens
the order under the cursor — its ticket, with the turn's numbers; `l` the
legend, `?` the About card, `m` turns the motion off. The vocabulary is
glossed on the screen itself (`SEAR ▐███▌ prefill`, `REHEAT 41 hit`) and
on its legend line; the numbers are the turn line's, in the terminal's own
foreground, and nothing paints over the terminal's background. It reconnects
with backoff if the daemon restarts and says so if none is running. It is
the one command in sous that imports [Textual](https://textual.textualize.io);
nothing else (the daemon included) loads it. It needs a terminal: piped or
run over a session with no pty, it prints a message on stderr and exits 2
instead of painting the pass into a pipe and waiting for a key that cannot
come.

**`sous statusline`** prints one line for Claude Code's `statusLine`
setting — `sous: decode 612 tok · 14.7 tok/s · eta 18s` during a turn,
`sous: idle · 5 slots · held` between turns, `sous: daemon down` when
nothing answers — and reads (and ignores) the JSON Claude Code pipes to it.
Add to `~/.claude/settings.json`:

```json
{"statusLine": {"type": "command", "command": "sous statusline", "refreshInterval": 1}}
```

`refreshInterval` matters: without it Claude Code re-runs the command only on
session events — a new assistant message, a compaction, a mode change — so the
line goes quiet for a whole subagent turn; with it the command runs every
second as well. The command loads none of the heavy dependencies (no
Textual, httpx or psutil); it drains stdin and fetches the status on a
thread joined to a half-second budget, printing `sous: daemon down` if that
runs out, so it costs the status bar nothing.

To bracket a subagent from the outside as well, a `SubagentStart` /
`SubagentStop` hook can append its own record — `{ts, agent_id,
agent_type}` — to a file; it sees the subagent's start and end, not the
progress-summary calls Claude Code makes in between, which only the turn
line and `sous top` show.

Forwarding is a plain HTTP/1.1 pass-through to `[gateway].upstream_url`
(default `https://api.anthropic.com`): the request body goes up byte for
byte; `Authorization`, `anthropic-beta`, `anthropic-version` and every header
sous does not recognise travel unmodified; only `Host` and the hop-by-hop
headers change — and, on the two Messages routes whose body sous had to read,
`Content-Length` is recomputed from the exact bytes — and responses carry the
upstream's headers minus the
hop-by-hop set, plus a `Via: 1.1 sous`, with `Date` and `Server` being the
daemon's own rather than the upstream's. sous stores no
credential — it never sees a token it did not receive to forward — honours
no `HTTPS_PROXY` or `~/.netrc`, follows no redirect, and retries nothing
(Claude Code retries). When the upstream is unreachable, forwarded requests
get a `502` (`504` on timeout) and the local model keeps working. WebSocket
features (voice) are not forwarded.

A whole-session-local run — every tier pinned to `sous-local` — remains
possible for exercising the endpoint; the recipe is in
[CONTRIBUTING.md](CONTRIBUTING.md#verifying-the-gateway-endpoint). It is a
verification setup, not a mode: it is exactly the trade "Why" and "How sous
compares" above argue against.

Claude Code executes the tool calls itself, with its usual permission
prompts; the local model only decides what to call. What a locally served
turn gives up, stated plainly:

- **No sandbox.** The gateway returns `tool_use` blocks and never runs a
  tool; `toolexec.py` (path confinement, allowlist, audit) is not in this
  path. Claude Code's own permission system is the boundary, and a 27B model
  inherits whatever permissiveness you configured for frontier subagents.
- **No Anthropic server-side or built-in tool types (no client-supplied
  schema).** `WebSearch`, `WebFetch`-as-server-tool and code execution run
  inside Anthropic's API; `bash_*`, `text_editor_*` and the other built-ins
  run on the client but arrive with their schema implied by the type. Both are
  dropped from a locally served request (logged as `dropped N tool(s) with no
  client-supplied schema`) — the local chat template can only offer a tool it
  has an explicit schema for. Claude Code sends custom-typed equivalents when
  it drives a non-claude model, so a local subagent keeps its file and shell
  tools. The main loop, forwarded upstream, keeps everything.
- **No thinking, no request-level sampling.** `thinking`, `temperature`,
  `top_p`, `top_k`, `stop_sequences` and `tool_choice` are accepted and
  ignored; the daemon's `[model]` sampler applies. Images and documents in
  messages become a one-line `[image omitted: sous serves text only]`
  placeholder.
- **One turn at a time; keyed prompt-cache slots.** Local turns are serialized
  behind the same lock as delegated tasks. The prompt cache keeps one slot per
  resident conversation (bounded by `[model].prompt_cache_gb`), so a subagent's
  consecutive turns reuse their own slot — copied and left in place when the
  budget can hold the copy, so a conversation that Claude Code branches (its
  progress-summary calls for a background agent take the same prefix down a
  different last turn) still finds it, and moved into the extending turn when
  it cannot, which is every hit at `prompt_cache_gb = 0`; a linear
  conversation holds at most its current and previous lengths — two subagents
  interleaving reuse theirs, and a delegated task running in between no longer
  wipes the gateway's slots the way a shared single slot did. Reuse stays
  owner-scoped — a slot is used only by the thread that built it, so a
  delegated task can never adopt a gateway slot or vice versa — but the
  budget and memory-pressure eviction below apply across the daemon, so a
  delegated task's own slot can displace a least-recently-used gateway one
  (and at `prompt_cache_gb = 0`, where only one slot fits, it will).
  A new subagent starts from a *fork* when it can: a copy of an earlier
  conversation's cache taken at a boundary its own prompt shares. Two
  boundaries are kept, each when it is long enough to clear the 4096-token
  floor. The *tools* boundary is where the template's tool block ends —
  Qwen3.5/3.8 render the `# Tools` block *before* the client's system text,
  and Claude Code's tool array is byte-identical across sessions and
  projects for a given subagent type (Explore, general-purpose, …), so a
  new `claude` process's first subagent turn starts ~45–56K tokens warm
  instead of paying ~170 s cold. The *header* boundary is where the whole
  system block ends, so a same-type subagent inside the same session starts
  ~57K tokens warm and prefills only its own brief. Each fork is a full copy
  of the KV at its boundary (~3.5 GiB at 57K tokens on the default model):
  one tools fork per tool set, plus one header fork per session that has
  used it, so several live `claude` sessions accumulate more than the
  tool-set count alone suggests. Budgeted by `[model].prompt_cache_gb`
  below. Forks live as long as the weights: `[model].idle_unload_minutes`
  drops them with the model.
  Two subagents still run one at a time;
  batching is a later phase.
- **Usage is split the way Anthropic's is.** `cache_read_input_tokens` is what
  the turn served from a resident cache slot and `input_tokens` the rest, so a
  warm subagent turn shows a few hundred input tokens and ~57K cache reads.
  `message_start` carries the whole count (it is sent before the cache
  decision); `message_delta` and the non-streaming body carry the split,
  which is where the SDKs read the input-side fields from when present. No
  `cache_creation_input_tokens`: every prompt stays resident, so it would only
  double-count the uncached tokens.
- **Mid-conversation system messages become `<system-reminder>` blocks.**
  Claude Code delivers attachments that arrive after the first turn (agent
  listings, MCP instructions, deferred tools) as a `role: "system"` message
  after the preceding user message. The chat template takes one system turn,
  at index 0, so the gateway renders such a message the way Claude Code's own
  fallback for models without the feature does — a `<system-reminder>` block
  in the user turn before it (after that turn's tool results). The model sees
  the same text in the same place a frontier model without the feature would;
  nothing the model needs is lost, and the conversation stays a strict extension of the
  previous turn's, which is what keeps it warm.
- **A shorter window, and Claude Code compacts inside it.** A frontier
  subagent has a 200K-token window; a local one has
  `[gateway].max_context_tokens` (131072 by default), which `sous claude`
  passes as `CLAUDE_CODE_MAX_CONTEXT_TOKENS`. Claude Code auto-compacts a
  conversation when *its own* token count reaches the window minus its
  output reserve (20K) minus a 13K buffer — 98K at the default window — and
  its "precompute" variant fires earlier still, at a fraction of that it
  takes from remote configuration. In practice a general-purpose subagent
  compacts after four or five tool turns at 131072: one call to write the
  summary (a warm hit, but a ~3.4K-token generation at the model's decode
  speed — about 3.5 minutes on the default model) and one to continue from
  it (the summary plus every re-attached file, ~30K tokens of prefill from
  the tools fork), and the agent then reports from a summary of itself.
  Nothing on the sous side changes this — both calls are served as cheaply
  as their content allows — and none of Claude Code's compaction switches
  (`CLAUDE_CODE_AUTO_COMPACT_WINDOW`, `DISABLE_AUTO_COMPACT`,
  `autoCompactEnabled`) is per-model: each would also change the frontier
  main loop, so `sous claude` sets none of them. The lever is the window:
  the default model's native context is 262144, and `max_context_tokens =
  262144` moves the classic threshold to ~229K at the cost of one more
  window of KV reserved out of the auto prompt-cache budget (8 GiB on the
  default model — the ~27 GiB auto budget described under `prompt_cache_gb`
  below becomes ~19 GiB on a 64 GB machine, still room for a tools fork
  beside a retaining conversation). Where the
  earlier precompute trigger lands at that window is not known until
  measured: its fraction comes from Claude Code's remote configuration,
  keyed by window size. Restart the daemon after the edit; `sous claude`
  passes the running daemon's value.
- **A client that disconnects does not stop the model.** A local turn runs to
  completion (so the next request never waits on a wedged lock); aborting
  mid-generation comes with batching, later. A forwarded stream, by contrast,
  is closed upstream the moment the client hangs up.

Each `/v1/messages` turn served locally logs one metadata-only line, for
example (wrapped here):

```
2026-09-10T19:26:14.025Z INFO sous.gateway: POST /v1/messages id=msg_… model=sous-local
  stream=1 status=200 input_tokens=84335 output_tokens=2887 stop=end_turn
  cache=hit took=turn@82647 reused_tokens=82647 prefilled_tokens=1681 forks=0 evicted=1 pressure=0
  load_s=0.0 queue_s=2.5 engine_wait_s=0.0 tokenize_s=1.1 ttft_s=9.8 prefill_s=7.9 decode_s=196.6
  prefill_tps=212.8 decode_tps=14.7 seconds=207.4 tools=1b21cd75 system=9f8e7d6c
```

`id` is the response's message id (the same one in `message_start`, so the
line joins to Claude Code's transcript). `cache` is `hit` (this
conversation's own slot), `fork` (a copy of a shared boundary — ~45–56K
reused tokens is a tools fork, ~57K a header fork) or `miss`; `took` names
the slot and its length — `turn@N` for this conversation's own slot copied
and left in place, `turn-moved@N` for one the budget could not hold a copy
of, or whose copy failed (removed and extended in place — every hit at
`prompt_cache_gb = 0`), `fork@N` for a shared boundary, `none` on a miss, and
on a hit whose warm attempt failed and was rebuilt cold (a
`WARNING py.warnings: … retrying cold` line comes first), where
`prefilled_tokens` and the phases below describe that cold attempt.
`prefilled_tokens` is what the turn had to prefill;
`forks`/`evicted`/`pressure` are what it published and what was dropped under
it — `evicted` counts every drop — budget, pressure, or the lengths below
the slot a turn takes, which the take retires — and `pressure` is the
subset of those `evicted` the pressure valve forced (not a second, disjoint
count). `load_s` (a model load, ≈0 when resident — including one this turn
only waited out, started by another request), `queue_s` (the wait for the
gateway lock — Claude Code's small background calls hold it too),
`engine_wait_s` (the wait for the engine itself, which a delegated task's
generation holds — part of `seconds` and `ttft_s`, in no phase below),
`tokenize_s`, `prefill_s` and `decode_s` say where the time went, as far as
the prompt cache measures it — not a strict partition of `seconds`: a turn
that starts from a copied fork slot times that copy into neither phase, so
the phases can sum to less than the total. `ttft_s` is not one more slice
alongside them: it spans from the start of generation to the first token,
overlapping `prefill_s` and however much of `decode_s` ran before that token,
so adding every field this way can just as easily run past `seconds` as fall
short of it. `prefill_tps`/`decode_tps` are that same phase attribution
expressed as a rate, not a throughput measurement — `prefill_s` also carries
fork copies and the snapshot, so a turn that publishes forks reports a lower
`prefill_tps` than its real prefill speed. `seconds` is the total, running
from the moment the turn takes the gateway lock, so client-visible latency is
`seconds` plus `queue_s`. On a miss the line adds `lcp=` (how many leading
tokens the render shared with the closest resident slot), `lcp_region=`
(`tools` below the tools boundary — the tool array changed; `system` below
the header — the system text did; `tools-or-system` below the header when it
is the only boundary, since a tool block under the floor, or one a template
renders inside the system block, cannot be told apart from the text there;
`conversation` otherwise; `-` when the probe found no boundary at all, whether
for lack of fork budget or because nothing cleared the 4096-token floor, or
when the session held no slot to compare with, `lcp=0`) and
`bounds=[tools,header]`, the boundaries the probe verified — either one is
left out when it never cleared the 4096-token floor, so this can also read
`bounds=[57123]` (only the header boundary) or `bounds=[]` (no probe ran). `tools=` and `system=` are
8-hex-character hashes of the rendered tool array and system text: comparable
across lines, not reversible. Refused requests log the same way at `WARNING`
with `status=` and `seconds=` — but there `seconds` is measured from
request receipt, not from the gateway lock, so it already *is*
client-visible latency, with nothing to add. A streaming turn that fails
after its SSE headers already went out logs `status=200` — the status the
client actually received — with an `error=` naming the failure; alerting on
`status=5..` alone misses these, so watch `error=` on streamed turns too.
Once a request parses, its `529`, `499` and failure lines carry `id=` too —
on a streamed failure, the id `message_start` already delivered. A client that
disconnects mid-turn does not stop the turn, and its line is still written
once the turn drains. A client that disconnects while its turn is still
queued behind another logs `status=499 error=abandoned` the same way — a
*local* 499, distinct from
the forwarder's own synthesized `499` for a client gone mid-forward
(below). A locally served `count_tokens` logs its `input_tokens`, `load_s`
(the model load it paid for), `count_s` (time inside the runner) and
`seconds` (client-visible, from request receipt — it includes any wait for
a free worker that `count_s` does not); the engine logs `model_load
seconds=N.N model=<model_id>` when it loads. One more line names the
Anthropic tool *types* a turn dropped, when any.
Each forwarded request logs one line too: `upstream`, method, path, the
model id when the body named one, the upstream's status, and seconds to
its headers — at `INFO` whatever the status, since that is the upstream's
verdict; a `502`/`504`/`499` the forwarder itself produced (unreachable
upstream, a timeout, a client gone) logs at `ERROR` or `WARNING` instead.
The daemon also disables uvicorn's access log, which would otherwise print
every request target — query string included — at INFO. Nothing else is
logged — not a request body, not a header value, not a query string, not a
response — at any level. Errors sous produces itself are Anthropic-shaped
(`{"type": "error", "error": {"type": ..., "message": ...}}`): an oversized
body (over 32 MiB) on `/v1/messages` is a `413 request_too_large`, a prompt
that fills the local window an `invalid_request_error` saying `prompt is
too long`, an unreachable upstream a `502 api_error`. Errors from the real
API come back exactly as it sent them.

## Configuration — `~/.sous/config.toml`

```toml
[server]
port = 8383

[model]
id = "mlx-community/Qwen3.8-27B-4bit"
idle_unload_minutes = 30   # a live `sous claude` session pins the model; the clock
                           # restarts when its last one exits
max_context_tokens = 32768
prompt_cache = true
# Cache slots kept resident beyond the running turn: "auto" sizes them
# from free Metal memory, a number sets the GiB, 0 keeps a single slot.
prompt_cache_gb = "auto"
temperature = 0.7
top_p = 0.8
top_k = 20
# Speculative decoding: ~1.8x decode on the default model with the shipped
# sampling, up to ~2.4x greedy. "" disables it. Block size 3 measured best
# on an M5 Pro (+3% on prose, +13% on code re-emission over the drafter's
# adaptive policy); 0 lets that policy pick the depth; anything above 5 is
# clamped, because mlx's fused attention kernel takes at most 5 verify rows
# on this model and 6–8 rows run 5–6x slower per layer. Auto-disables with
# a warning when the drafter can't serve the configured model.
speculative_draft_id = "z-lab/Qwen3.8-27B-DFlash2"
speculative_block_size = 3
# INT8-activation prefill on the M5 tensor units (M5-family or newer, macOS
# 26.2+; warns once and prefills stock elsewhere). ~1.4x prefill on the default
# model, but int8 activations change numerics — measured drift is inside what
# 4-bit weights already add — so it stays off until a tool-loop A/B says
# otherwise.
int8_prefill = false

[budgets]
max_turns = 40
max_minutes = 15
max_tokens_per_generation = 4096

[commands]
allowlist = ["pytest", "python -m pytest", "npm test", "npx eslint",
             "npx prettier", "ruff", "black", "mypy", "go test",
             "cargo test", "cargo check", "make test", "uv run pytest",
             "uv run python -m pytest", "uv run ruff", "uv run black",
             "uv run mypy", "uv run ty"]
timeout_seconds = 120
approval_timeout_minutes = 10

[context]
mode = "fixed"     # "auto": size the window per task from free memory
fraction = 0.8     # auto: share of remaining memory headroom the KV cache may use
min_tokens = 8192  # auto: never shrink the window below this

[tasks]
retention = 200

[gateway]
# EXPERIMENTAL — see "Gateway mode" above. Serve Claude Code subagents from the
# local model and forward everything else to the real API, on the same port.
enabled = false
local_models = ["sous-local"]   # model ids served locally; every other id is forwarded upstream.
                                # Never claude-*: Claude Code ignores its context-window
                                # env vars for those ids (the config rejects them).
upstream_url = "https://api.anthropic.com"  # where non-local requests go: an https origin, no path,
                                            # ASCII hostname or IP literal.
                                            # Plain http is accepted for a loopback host only.
max_context_tokens = 131072     # server-side limit on prompt + reply tokens for local turns;
                                # a Claude Code subagent's prompt — its agent prompt plus the
                                # session's tool schemas — is ~50-58K tokens before it does
                                # anything, and it asks for 32K of output; 65536 was too small
                                # in practice. Positive values below 49152 are raised to it.
                                # `sous claude` sets CLAUDE_CODE_MAX_CONTEXT_TOKENS to the
                                # running daemon's value of this (restart it after an edit);
                                # without the launcher, set it yourself or a long subagent
                                # conversation grows past it and fails with "prompt is too long".
                                # Claude Code auto-compacts a local subagent once its own
                                # count nears this minus ~33K (sooner with its precompute
                                # trigger); 262144 — the default model's native length —
                                # pushes that out at the cost of 8 GiB of prompt-cache budget.
generation_timeout_minutes = 30
```

Every value is optional; the allowlist is re-read on every command execution,
so edits apply instantly. Swap `[model].id` for any MLX text or vision model
(e.g. the `-8bit`/`-4bit` conversions, or a fast MoE coder via mlx-lm).

`[context] mode = "auto"` sizes the worker's context window per task instead
of using the fixed `[model].max_context_tokens`: when a task starts, sous
measures the remaining memory headroom (the tighter of the Metal working-set
ceiling and available system RAM), lets the KV cache have `fraction` of it,
and clamps the result between `min_tokens` and the model's native maximum.
The window is a cap, not a reservation. With cache reuse on (the shipped
default), the KV cache lives for the whole task, so `fraction` bounds
sustained residency, and a `run_command` subprocess competes with a live
cache that used to be freed between turns. With
`[model].prompt_cache = false` it bounds only a per-generation peak.
Residency still tracks the tokens a task actually uses, not the window.
Every task's report records the window it ran with and why
(`budget.context_tokens` / `budget.context_reason`). The default model's
hybrid attention makes context unusually cheap (only 16 of its 64 layers
accumulate KV — about 64 KiB per token), so an otherwise-idle 64 GB machine
gets the full native 262k window. If sizing fails for any reason, the task
runs with the fixed `max_context_tokens` and a warning.
Cache reuse pays most in `auto` mode: since elision is the only thing that
discards the cache, and elision fires only when the prompt exceeds the window,
a window the task never reaches means the cache survives the whole task. The
shipped default is `fixed` at 32768 tokens.

`[model].prompt_cache` (default `true`) reuses a KV cache across the turns of a
conversation, prefilling only what the conversation gained instead of the
whole thing every turn. All of a task's generations run on one worker-owned
thread so the cache survives between turns; measured on the default model in
one process, six growing turns took 29.5s warm against 77s cold, with per-turn
time flat instead of growing. Set it to `false` to prefill every turn from
scratch.

`[model].prompt_cache_gb` (default `"auto"`) bounds the caches kept resident
*beyond* the turn that is running: each conversation's current length, its
previous one when the budget holds both (see below), plus *fork*
slots at every boundary long enough to be worth copying (4096 tokens or
more): one at the end of the tool block, shared by every Claude Code session
and project that presents the same tool array, and one at the end of the
whole system block, shared by same-type subagents of one session (see
above). Prefixes must match token for token — one added, removed or
reordered tool is a different tool set with its own tools fork and header forks. Each fork
is a copy of the KV at its boundary, ~3.4–3.6 GiB at ~57K tokens on the
default model: one tools fork per tool set, plus one header fork per
session that has used it, so a daemon that has seen the usual three tool
sets across one or two live `claude` sessions holds roughly five to nine
forks, ~17–31 GiB (the tools forks stay most-recently-used, since every new
session touches them). `"auto"` is what Metal's recommended working set has
left once the weights, one full context window of KV (the larger of
`[model]`'s and `[gateway]`'s) and 2 GiB of slack are paid for — about
27 GiB on a 64 GB machine with the default model and gateway window, room
for those forks and several conversations; each live conversation under the
auto budget also keeps its previous length resident (the slot a branch of it
starts from), so a live conversation is two slots, ~8 GiB at 63K tokens,
beyond the forks; a 48 GB machine should set it to `0` (forks off, one slot), or to
more than twice one conversation slot at the length its conversations reach
— a little over 8 GiB with the default model at ~57K tokens — because the
copy that keeps a conversation's previous length resident is taken only
when that length and the longer one the turn publishes both fit; below that
every hit moves its slot, and a branch of the conversation (a
progress-summary call) starts from the fork instead. The copy's room comes
from least-recently-used slots,
forks included, so keeping a fork resident beside a retaining conversation
wants about four slots' worth, ~14 GiB. A value below ~14 GiB makes every
cold turn take a fork copy that a later turn then evicts to make room for
its own copy, so it pays the fork and never reuses it. Slots are
evicted least-recently-used first when the budget, a count of 16, or memory
pressure says so. Pressure is two readings: Metal's own headroom (room for one
more window of KV), and the kernel's memory-pressure level — at *warn* each
publish drops one least-recently-used slot (a cold turn publishes up to three
times: two forks and its turn slot; a warm turn once), at *critical* every
slot but the one that just ran. The
kernel's level is used rather than free RAM because a freshly loaded model
leaves ~17 GB of its weight files in the page cache, which reads as "used"
for a while and would evict the forks a cold turn had just made. The
conversation that just ran is never evicted by
its own turn, and a cold turn's second fork copy never evicts its first, so
`0` means exactly one slot (the pre-3a behaviour) and a 32 GB machine
degrades to that on its own. Forks live as long as the weights do:
`idle_unload_minutes` drops them with
the model, so "every new session" means every new session inside that
window.
`server_status` (and `GET /sous/status`, the same document over HTTP, plus
the recent turns and tasks) reports the engine — `holders` (live `sous
claude` sessions pinning the model), `loading` (a load in progress, a
preload included), `memory_gb` and `prompt_cache` — slots, resident bytes,
hits, fork hits, retained and moved turn-slot takes, evictions and the
subset the pressure valve took — counts only — and `inflight`, the turn
being served right now with its phase, tokens and rate.

`[model].int8_prefill` (default `false`) runs the prefill matmuls as INT8 activations
against the checkpoint's packed 4-bit weights on the M5 GPU's neural accelerators
(Apache-2.0 kernel derived from oMLX, compiled at model load — no build step). Measured
on an M5 Pro with the default model: 492 → 695 tok/s at 4K tokens, 410 → 570 tok/s at
32K (MLP and linear-attention projections; attention projections are not routed, see
#76). Decode and speculative verify are untouched. It changes prefill numerics (KL 0.033
vs the stock path on a code prompt; 4-bit weights alone are 0.052 vs 8-bit), which is
why it ships off. Needs an M5-family or newer GPU and macOS 26.2+; anywhere else the
`server_status` tool reports `int8_prefill: unavailable` with the reason and prefill
runs stock. Only dense Qwen3.5-family models (`model_type` `qwen3_5`, as the default
model is) with affine 4-bit, group-size-64 weights route, and the MoE variant is refused;
a checkpoint with no eligible projection warns once; in a mixed checkpoint, ineligible
projections fall through per projection.

`temperature`/`top_p`/`top_k` control the worker's sampler (Qwen's own
documented non-thinking-mode defaults). Greedy decoding (temperature 0)
sounds safer but isn't: it gives the model no way to escape a bad
completion once it happens, since a near-identical prompt plus a nudge
still argmaxes to the same wrong output every time.

## Smaller machines

The default model fits 64 GB and 32 GB machines. The alternative below shares
the default's `qwen3_5` architecture, so it loads through the exact same
mlx-vlm path — edit `[model].id` in `~/.sous/config.toml` and the next
delegation downloads and uses it.

| Unified memory | `[model].id` | Weights |
|---|---|---|
| 64 GB / 32 GB (default) | `mlx-community/Qwen3.8-27B-4bit` | ~16.1 GB |
| 16 GB | `mlx-community/Qwen3.5-9B-MLX-4bit` | ~6 GB |

The default is the affine 4-bit Qwen3.8-27B: half the footprint of the
8-bit-class quants with no measured tool-loop quality loss. It also keeps
mlx-vlm's speculative-decoding fast path available, which requires affine
quantization (4-, 5-, or 8-bit); mxfp quants fall into a much slower
per-token verify fallback (krcm0209/sous#58 has the measurements behind
both claims). The 16 GB pick
drops to the 9B tier because an 8-bit 9B (~11 GB) would crowd the ≈10.7 GB
Metal working-set limit of a 16 GB machine once the KV cache lands on top of
the weights; on 16 GB, also consider `[model].max_context_tokens = 16384` if
you see memory pressure. (`mlx-community/Qwen3.5-9B-mxfp8`/`-mxfp4` look
like the obvious picks, but as of 2026-08 they are empty placeholder repos
with no weights.)

The 16 GB alternative passed the same worker-path validation as the default
(see [Validation status](#validation-status)), run on the 64 GB test
machine — which validates the models and quants through sous's whole stack,
not the memory fit on physical 32 GB / 16 GB hardware (that remains
arithmetic: weights plus KV-cache headroom). Smaller workers still fail more
tasks in general and make reviewing the diff matter more — but a reviewed
draft from a small local model costs your plan nothing.

## Security model

- Workers are confined to the `project_root` of their task (symlink-resolved;
  `.git/` writes denied). The sous control directory (`~/.sous/` — config,
  allowlist, task db, transcripts) is never writable from inside the sandbox,
  and a `project_root` that contains it is rejected outright.
- No shell: commands run as argv (never through a shell), with an
  environment scrubbed down to `PATH`, `HOME`, `LANG`, `LC_ALL`, `TERM`,
  `TMPDIR` (anything else, including `*_TOKEN`/`*_KEY`/`*_SECRET`/`*_PASSWORD`
  vars, is stripped). Only allowlisted commands run without approval.
  `PATH` itself is adopted from your login shell once at daemon startup, so
  allowlisted commands resolve exactly as they do in your terminal no matter
  how the daemon was launched (launchd starts agents with the bare system
  `PATH`, which would otherwise turn every `uv run ...` into an approval).
- Non-allowlisted commands pause the task for explicit human approval
  (auto-deny after `approval_timeout_minutes`).
- Path confinement bounds the worker's *edits*, not what an allowlisted
  command can do. Allowlisting a command that executes repo-resident code —
  any test runner (`pytest`, `npm test`, `make test`, ...) — is equivalent
  to granting arbitrary local code execution over code the worker just
  wrote: the worker can write a `conftest.py` or test file that does
  anything the command's process can (network egress, reading any
  user-readable file), and the allowlisted verify run executes it without
  approval. Calibrate the allowlist accordingly, and review diffs before
  trusting verify output.
- There is no network sandbox: an allowlisted or human-approved command can
  still reach the network if the command itself does (e.g. `npm test`
  hitting a registry). Keep the allowlist narrow and review approval
  requests before saying yes.
- Command timeouts kill the command's whole process group (SIGTERM, short
  grace, SIGKILL) before file changes are audited — but a descendant that
  double-forks and calls `setsid()` escapes into a new session and survives
  the group kill. Closing that residual requires cgroup/OS-level confinement
  that macOS does not offer.
- The before/after file audit around each command is stat-based:
  `(mtime_ns, size, ctime_ns)` per file. mtime and size alone are forgeable
  by code the command executes (equal-length rewrite + `os.utime` restore);
  ctime is what makes the audit tamper-resistant, because no userspace API
  can set it and `os.utime` itself bumps it. That resistance has limits: a
  process running as root (e.g. via a mount trick or raw-device write) or
  manipulation of the system clock between the two snapshots could still
  hide a change. The audit is a safety net against the sandboxed worker and
  the code it runs — not against a privileged attacker.
- Every worker turn is journaled to `~/.sous/tasks/<id>/transcript.jsonl`.
- The MCP endpoint binds to 127.0.0.1 only.
- **Gateway mode bypasses the sandbox by design.** A locally served Claude
  Code turn never touches `toolexec.py`: the gateway hands `tool_use` blocks
  back and Claude Code executes them under its own permission rules. The
  gateway binds to `127.0.0.1` only and refuses foreign `Host`/`Origin`
  values on every route, forwarded ones included. It forwards the
  `Authorization` header Claude Code sends with every request to
  `[gateway].upstream_url` unmodified and nowhere else, stores it nowhere,
  adds no credential of its own (no `~/.netrc`, no proxy environment), and
  never logs a request body, header value or query string. A plain-`http`
  upstream is accepted for a loopback host only. It is off by default.

## Validation status

Validated end to end on an M5 Pro / 64 GB (originally against
`mlx-community/Qwen3.8-27B-mxfp8`, which remains a supported `[model].id`):

- **Worker path** — a delegated "add type hints and docstrings" task completed
  in 57s over 3 turns (`done` / `completed`). The worker edited the file, chose
  to run `pytest` to check itself, and reported accurately; an independent
  re-run of the tests confirmed it.
- **MCP surface** — driven by a real MCP client over streamable HTTP, the same
  path Claude Code uses: all six tools registered with the expected names, and
  a delegated task ran to `done` / `completed` in 42s with a correct diff and
  verify output.
- **Current default (`Qwen3.8-27B-4bit`)** — three delegated tasks
  (module-from-spec, docstring sweep, test scaffolding) through the real
  worker loop on 2026-08-29, all `done` / `completed` in 4/7/3 turns; every
  artifact passed independent grading, including hidden spec tests
  (krcm0209/sous#58).
- **Other models through the same stack** — the same worker-path check also
  passed on `Qwen3.8-27B-mxfp4` in 35s over 4 turns — confirming mlx-vlm
  loads mxfp4-mode quants — and on the 16 GB pick `Qwen3.5-9B-MLX-4bit` in
  25s over 8 turns.

Tool-call parsing accepts both the XML-ish `<function=…>/<parameter=…>` format
that Qwen3 emits and the hermes JSON format used by other MLX models.

## Limitations

- MLX generation cannot be aborted mid-stream. Generations are serialized by
  a per-engine lock (and the engine is never idle-unloaded while one is in
  flight), so a truly wedged generation delays subsequent tasks until the
  daemon is restarted. Running the worker in a separate process (process
  isolation) is the future fix.
- `scripts/e2e_smoke.py` uses a 0.6B model so it stays cheap to run. That model
  is too small to reliably emit a `finish` call, so the script usually ends
  `failed` or `budget-exhausted` even when it writes the right file — it
  exercises the plumbing, not model competence. The real-model runs above are
  the meaningful end-to-end evidence.
- Gateway mode (experimental) serves one local turn at a time on a
  keyed prompt cache (a conversation's last two lengths plus tools and header forks, budgeted
  by `[model].prompt_cache_gb`), drops Anthropic server-side and built-in tool
  types from local turns (the ones that carry no client-supplied schema),
  ignores request-level sampling and thinking, and finishes a local turn
  even after the client hangs up. It serves the model ids in
  `[gateway].local_models` locally and forwards everything else to
  `[gateway].upstream_url` over HTTP/1.1 only — no WebSocket upgrade, no
  HTTP/2, no proxy environment.

## Development

Setup, the checks CI runs, and the pull-request process are in
[CONTRIBUTING.md](CONTRIBUTING.md).

Manual E2E with the real model: `sous serve`, register with `claude mcp add`,
then ask Claude to delegate something trivial and watch `sous status`.

Design spec: `docs/superpowers/specs/2026-08-14-sous-design.md`.
