Metadata-Version: 2.5
Name: litmus-ai-sdk
Version: 0.3.2
Summary: The Litmus client: probe a tool surface, ship the evidence, get a faithful mock back.
Author: Litmus
Keywords: agents,evaluation,mcp,sandbox,tool-use
Requires-Python: >=3.11
Description-Content-Type: text/markdown

# `litmus` — the CLI, and how a repo talks to a workspace

One file, `cli/litmus.py`, stdlib only. It is the whole client. Everything it
does is an HTTP call you could make by hand — the CLI is a convenience, never a
requirement.

## Getting it (three shapes, one file)

| Shape | Command | When it wins |
|---|---|---|
| **Installed** | `pip install litmus-ai-sdk` → `litmus-ai …` | A customer's agent bootstrapping inside their own repo, and anyone who runs it daily. Names no infrastructure, so it can be pasted into a prompt that goes to someone we do not host. |
| **Served by the workspace** | `curl -sO <workspace>/litmus.py` | No registry, no install, and the client always matches the workspace that served it. Works from a cloud dev session, a container, an air-gapped-ish laptop, or a machine with no PyPI reachability. |
| **No client at all** | `curl -X POST <workspace>/v0/evidence …` | CI, another language, a script. The API is the contract. |

The first two are **the same bytes**. The wheel does not contain a rendering of
`cli/litmus.py`; it contains that file, renamed on the way into the wheel so it
imports as `litmus_ai_sdk` (`pyproject.toml`, `force-include`). So the digest
`litmus-ai --version` prints is the digest `GET /litmus.py.sha256` publishes, a
customer can check one against the other, and the probe brief the console lifts
out of the served text cannot drift from the client the customer installs.

`GET /litmus.py` is deliberately unauthenticated: it serves client *code*, which
holds no secrets. The token is what is protected, and it never appears here.

## The name

**`litmus-ai-sdk`** on PyPI. Sidharth's call, 2026-09-15, after weighing
`litmus-sdk` / `litmus-ai-sdk` / `litmussdk` / `litmusai` — and made because the
alternative was worse: the probe prompt was telling a customer's agent to `curl`
a script off a shared-workspace hostname built out of **another customer's**
project name — which reads, to the customer running the command, as an
unexplained third party inside their own install instructions. (The reasoning in
full, with the names, is internal: `DIRECTION.md` §5, 2026-09-15/16. This file is
the distribution's description on PyPI, so it names no customer.) A `pip install`
line names nobody's infrastructure. It is not up for revisiting.

Two names he did not specify, which are just as permanent, and why they are what
they are:

| | Name | Why |
|---|---|---|
| distribution | `litmus-ai-sdk` | His decision. Also the only one available: PyPI has long held `litmus` for an unrelated pytest-skeleton generator (1.0.1), so `pip install litmus` already gets a stranger's package. |
| import | `litmus_ai_sdk` | The PEP 503 normalisation of the distribution name, one-for-one. A reader who sees the import knows exactly what to `pip install`, and vice versa — the failure mode of a clever short import name is a support conversation that starts by working out which package it came from. It also cannot collide with the internal product's top-level `litmus` package, which matters below. |
| console script | `litmus-ai` | **Not `litmus`.** See the collision. |

### The collision, and why the console script is `litmus-ai`

This repo declares *two* distributions and they share a word:

- the root `pyproject.toml` — distribution `litmus`, package `litmus/`, console
  script `litmus`. The internal V1.1 product: `run`, `grade`, `analyze`,
  `export`, `compare`, `world`, `validate`, `provision`, `fixture`. Customers
  never install it and it is published nowhere.
- `cli/pyproject.toml` — distribution `litmus-ai-sdk`, module `litmus_ai_sdk`,
  console script `litmus-ai`. This file. What a customer installs.

If the second one also installed a script called `litmus`, then in any
environment holding both — which is every Litmus engineer who reproduces a
customer issue — `litmus` would be whichever was pip-installed most recently.
That is not a theoretical hazard, because **the two tools share the verbs
`export` and `compare`** with entirely different meanings, different flags and
different outputs. `litmus export` would quietly do the other thing, succeed,
and produce a wrong artifact. A name that can be shadowed is a name that will be.

So: one script, `litmus-ai`, which can neither shadow nor be shadowed, matches
the distribution name the reader just typed, and fails loudly (`command not
found`) for anyone following an older doc — which is the correct failure. Python
entry points are per-distribution and per-environment; nothing about `pip
install litmus-ai-sdk` can now reach the internal tool, in either direction.

Treat all three names as permanent: they go into customer prompts, agent
instructions and support transcripts, and the cost of changing one later is
every one of those.

## The version — what the number means and when it moves

`__version__` in `cli/litmus.py` is the only place the version is written;
`cli/pyproject.toml` reads it, so the wheel on PyPI, the module a customer
imports and the bytes the workspace serves at `GET /litmus.py` cannot claim
different versions of the same file.

**The scheme is `MAJOR.MINOR.PATCH`, and what it versions is the surface a
customer meets** — the verbs, their flags, and the briefs the client prints.
Not the internals: this file is one module, and "an internal refactor" is not a
category a customer can observe.

| bump | when |
|---|---|
| **PATCH** | behaviour or wording of something that already exists changes — a brief's text, an error message, a fix |
| **MINOR** | a verb, a brief or a flag is **added** |
| **MAJOR** | a verb, a brief or a flag is **removed** or its meaning changes |

**Every commit that changes the bytes of `cli/litmus.py` moves the number, in
that same commit.** Not at release time, not when someone decides a batch is
worth shipping. The reason is the rule this repo already holds one product
over: *a fix to a cached derived layer that does not bump the layer's version
has not shipped, it has only been written* (DIRECTION §5 (61), (64), (73)), and
the published wheel is exactly such a layer.

**This rule was already written here, in prose, on 2026-09-16, and it was
broken within three days.** `census` landed in `BRIEFS` on 2026-09-19 with
`__version__` still reading `0.1.0` and nothing uploaded; a MedScout probe agent
ran `litmus-ai brief census`, got `usage: litmus brief <probe>`, and found it
before we did. A rule nobody can hold is a rule nobody applies — so it is a test
now, not a paragraph. `tests/test_published_client_contract.py` reads the
release off PyPI and fails when these bytes differ from it under the same
number, and fails again when anything a customer is told to run is missing from
what was actually published.

Current: **0.3.2** — the PATCH number LIT-255 pinned for the hotfix, although
it also ADDS a flag (`--explain-drops`), which the table above would call a
MINOR; the next added verb or flag goes to 0.4.0 rather than compounding that.
The fix: a root `parent_id` that is a
SENTINEL (`undefined`, `null`, `none`, `nil`, `0` — Datadog LLM Observability
writes the first one) is now read as "no parent" rather than as a dangling id,
so an export whose roots say `undefined` converts instead of dropping whole
(0.3.1 read 64,605 spans of a real export and emitted nothing). Drops now
carry the SHAPE of the parent ids behind them — classes, lengths and counts,
never an id — and a root may be inferred, and says so on the trace, in exactly
one narrow case (LIT-255).

## Publishing

`.github/workflows/publish-cli.yml` builds and uploads on a push to `main` that
touches `cli/**`, and on manual dispatch. It authenticates to PyPI by **trusted
publishing** (OIDC), so there is no API token in GitHub secrets, and it no-ops
until the repo variable `PYPI_PUBLISH_ARMED` is set — merging it ahead of the
PyPI-side setup is safe. It refuses to upload a version that is already on the
index, so a `cli/README.md`-only change is a no-op rather than a failure.

By hand, if you ever need to — publishing is outward-facing and irreversible, a
PyPI name and version can never be reused:

```bash
cd cli
uv build --out-dir dist/                       # or: python -m build
python -m twine upload dist/*                  # needs a token
```

`twine` reads `TWINE_USERNAME` (the literal `__token__`) and **`TWINE_PASSWORD`**
— an env var *name*; the value is a PyPI API token that lives in the secret
store and never in a file, a log, a commit or a URL. That token is what trusted
publishing exists to retire: it is long-lived, it is copyable, and it is not
scoped to a commit.

**Editing `litmus.py` is a release, not a commit.** The console's probe
prompt tells every customer to hold `litmus-ai --version` against the digest
their workspace publishes at `GET /litmus.py.sha256`, and that comparison is
only true while the published wheel and the deployed file are the same bytes.
A merge to `main` redeploys every workspace within minutes, so a change here
that ships without a matching version bump and upload leaves the workspace
serving bytes no installed client matches — and the console has already told
the customer to compare two numbers that will not agree. So, in one commit:
change `litmus.py` and bump `__version__`; the publish workflow does the upload
on the merge. The prompt tells a customer what to do when the two differ
(`pip install -U litmus-ai-sdk`, or the workspace `curl`, which is that
workspace's own bytes by construction) — that is the recovery, not permission
to let them drift.

The console-side wiring this unblocked is done: `dashboard/lib/brief.ts`
renders `pip install litmus-ai-sdk` first, keeps the `curl` block as a named
fallback for a machine with no index, and spells every command `litmus-ai`.

**Verify the download.** A reviewer refused to run this file because it arrived
with "no checksum, no integrity guarantee" — correctly. The response carries an
`X-Litmus-SHA256` header, and `GET /litmus.py.sha256` returns the digest on its
own so it can be fetched by a different route than the code:

```bash
curl -sO <workspace>/litmus.py
curl -s <workspace>/litmus.py.sha256          # {"sha256": "…", "bytes": …}
shasum -a 256 litmus.py                       # compare
```

Until the workspace has TLS this detects tampering rather than preventing it —
anyone able to rewrite the body can rewrite the header. The digest is worth
having anyway (it pins exactly what you ran), and it is not a substitute for
either TLS or, better, an installed package with a pinned version.

**A CLI is not the non-technical surface, and pretending otherwise is how
products get built for the wrong person.** The non-technical path is the console
(`/console`) and the dashboard: paste a URL, read a report. What makes the CLI
usable by a non-expert is that *they never type it* — their coding agent does,
after reading `litmus brief probe`. The human types English.

## The loop

```bash
curl -sO http://<workspace>/litmus.py

# the token is a per-org API token from the console (Settings → Tokens, scope
# `push`); it pins the org, so `push` needs no --org. Never the service token.
python3 litmus.py init --url http://<workspace> --token litk_…

# 1. map a real tool surface (nothing leaves the machine yet)
python3 litmus.py probe --mcp "npx -y @acme/mcp-server"
python3 litmus.py probe --mcp "https://acme.example.com/mcp" --header "Authorization: Bearer …"
# hosted and OAuth-protected? Nothing to configure: the 401 is discovered, a
# browser opens once, you sign in as yourself. docs/ops/sign-in-to-a-hosted-mcp.md
python3 litmus.py probe --mcp "https://acme.example.com/mcp"

# 2. ship the evidence
python3 litmus.py push .litmus/sessions/<session>.mcp.jsonl

# 3. get a mock back — compile + mint, one live MCP URL out
python3 litmus.py serve acme .litmus/sessions/*.mcp.jsonl

# 4. the question the product rests on: is the mock like the real thing?
python3 litmus.py compare --real "npx -y @acme/mcp-server" --mock <that URL>
```

Installed, the same loop is `litmus-ai init …`, `litmus-ai probe …` and so on —
identical bytes, identical verbs, just on `$PATH`. `litmus-ai --version` prints
the version, the file's sha256 (compare it with `GET /litmus.py.sha256`) and the
workspace it is pointed at; `litmus-ai status` is the one that asks the
workspace for *its* build version, because that is a fact about a different
machine and `--version` makes no network call.

Steps 1 and 4 need no Litmus account for the *real* half — `probe` writes local
files only. A workspace URL and token are needed from step 2 on. A per-org
token opens exactly one org: `push`, and reading that org's evidence, jobs and
pipeline back; used against another org it is refused with the mismatch named,
and a revoked or expired one is refused with the reason. `serve`, `shell` and
`status` still need the workspace's service token, which customers do not get.

## The verbs

- `probe --mcp <server cmd | url>` — mechanical sweep of every tool: required
  args filled from the schema, ids harvested from earlier replies and reused,
  optionals both supplied and omitted, and a deliberate bad id at the end
  because error envelopes are part of the surface. `--read-only` skips anything
  that does not look like a read.
- `call --mcp <…> <tool> --args '{…}'` — one recorded call. This is the verb an
  **agent** drives, thinking between calls; that is why the protocol in
  `litmus brief probe` beats an unattended sweep. Calls append to the same
  session file.
- **A server that answers 401 is signed in to, not given up on.** `probe` and
  `call` implement the MCP authorization specification: the challenge is read,
  the protected-resource and authorization-server metadata are discovered, this
  client registers itself (RFC 7591) where the server allows it, and the person
  signs in once in a browser with PKCE and a loopback redirect. The token is
  reused after that and lives in `~/.litmus-mcp-auth.json` at mode 600 — never
  in `.litmus/`, which is what `push` sends us. `--header` is unchanged and
  suppresses the flow (you supplied the credential); `--client-id` carries a
  client id from a vendor who does not register clients dynamically;
  `--no-auth` runs no flow at all, which is how a census records what the
  surface answers with NO credential. Failures name which of the five causes
  fired: `docs/ops/sign-in-to-a-hosted-mcp.md`.
- `record --wrap gh,gt -- <command>` — for surfaces that are CLIs, not MCP:
  shims wrap the real binaries and log argv/stdout/exit.
- `push <file>` — traces, tickets, workflow text, telemetry.
- `serve <name> <sessions…>` — compile through the replay gate and mint a world.
  A mock that cannot reproduce its own evidence refuses to serve, and says why.
- `compare --real <…> --mock <…>` — replay identical calls against both.
- `shell --bundle <name-or-dir> --mint` — materialize a world locally and
  PATH-shadow its fake binaries, so an agent runs against the mock unmodified.
  A **name** rather than a path is fetched from the workspace and cached in
  `.litmus/bundles/`, digest-checked. This is the path for agents whose surface
  is CLIs rather than MCP: `litmus shell --bundle <bundle-id> --mint` gives you
  working `linearis`, `huginn`, `gh` and `gt` on a machine where none of them
  are installed, and logs every invocation into the world.
- `brief probe` — the two-phase protocol, as instructions for someone else's
  agent: phase 1 maps the surface (schemas in and out, auth, errors, refusals,
  which tools write), phase 2 plays out whole jobs so the corpus carries user
  turns a task can quote. Intelligence flows down; evidence flows up. The
  fidelity 1.00 on Litmus's battery was measured on brief **v1**, which was
  phase 1 alone; v2 does not inherit it.
- `export --org <org> --surface <surface>` — assemble the thing a customer
  receives (a customer PRD's technical requirement 7): the pipeline's suite and
  mock byte-for-byte, the runner vendored as source with its dependencies
  pinned by hash, this file with its digest in the README, a config that
  names secrets by env var only, a manifest of every pin, and a `run.sh`
  that runs cells, grades them and renders a scorecard — with no workspace
  in the loop. It assembles; it authors nothing, and a pin the pipeline did
  not produce is `null` in the manifest with the reason. Reproducible: two
  exports differ only in the timestamp.
- `scorecard <results-dir>` — render a scorecard from stored runs alone: per
  cell n, the five terminal states with `infra_failure`/`invalid` shown and
  excluded, the key's facts one by one, each detector's named findings, and
  a tier and a path on every number. No bare ratio.

## How `compare` grades

Byte-identity is the wrong bar: a simulator that mints its own ids is still
faithful if the shape and the outcome hold. So each replayed call lands in one
of five buckets, worst-first in the report:

| Verdict | Meaning |
|---|---|
| `exact` | identical payloads |
| `shape_match` | same fields and types, different values (ids, timestamps) |
| `shape_mismatch` | the mock returned a different structure |
| `outcome_mismatch` | one succeeded where the other failed — the serious one |
| `missing_tool` | the real surface has it, the mock does not |

Plus a catalog diff: tools the mock **invented**, and tools it **missed**.
`--push` files the whole report in the workspace as evidence.

### One thing `compare` cannot do for you

If the replayed calls **mutate**, the two sides are not starting from the same
place. The mock gets a freshly minted world; the real surface is wherever your
last probe left it — cart already full, order already placed, the id you are
fetching now existing. Divergence then measures **state drift, not fidelity**,
and it will look like the mock is wrong when it is not.

So for a trace with mutations in it, one of these has to be true:

- the real surface is reset between the probe and the comparison (a fresh
  staging tenant, a seeded test account, a teardown script), or
- you compare on a read-only subset, or
- you read the result as a *state* diff and not a fidelity number.

There is no way for us to reset your system from here, so this is a fact about
the measurement rather than a bug we can close. `compare` reports what it saw;
knowing which world the real side was in is your half of the contract.
