Metadata-Version: 2.4
Name: llm-market
Version: 0.2.17
Summary: Open clearinghouse for LLM bids, councils, and evidence-backed routing.
Author: LLM Market contributors
License-Expression: MIT
Keywords: llm,routing,llm-market,openrouter,agents
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sentence-transformers>=5.0.0
Requires-Dist: torch>=2.0.0
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pandas>=2.0.0; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Provides-Extra: jepa
Dynamic: license-file

# LLM Market

LLM Market is an open, sequential router for LLM work. In Codex, a bundled
state/action JEPA chooses the next bounded compute action at each meaningful
event: accept, answer quickly, execute normally, plan, review or repair, rescue,
or run a small council.

The point is simple: stop sending every prompt to the most expensive model when a
cheaper route can match the answer.

## Current Behavior

Codex routing is automatic and JEPA-first after a one-time install. Users ask a
normal question; lifecycle hooks send local, transient input to a warm
controller, and Codex privately applies the selected next action. There is no
activation phrase, model chooser, execution contract, or per-turn setup.

The controller carries a six-state trajectory across prompts, tools,
validation, workers, follow-ups, and compaction. A conclusive validator pass
forces acceptance, a failure forbids acceptance, and hard limits prevent
concurrent mutators or runaway escalation. Missing infrastructure fails open to
ordinary Codex.

Standalone `plan` and `run` retain the general model-market surface for
provider experiments. The retired whole-route Codex allocator, sticky epochs,
and Gate 2 lifecycle are no longer shipped or callable.

Training runs, raw prompts, provider output, and local evidence stay outside the
package. The wheel contains only sanitized inference assets: the checkpoint,
local embedder, consequence heads, scalers, schema, thresholds, and provenance.

## Quickstart

For automatic Codex routing, hand Codex
[`docs/LLM_MARKET_INSTALL.md`](docs/LLM_MARKET_INSTALL.md). It installs the
exact current PyPI release into an isolated environment, installs the bundled
skill and agents, checks the integration, and runs the packaged hook-to-controller
smoke. No source checkout, personal code history, personal JEPA, API key, model
selection, or activation phrase is required.

For the standalone model-market CLI:

```bash
python -m pip install llm-market
llm-market plan "Should this prompt use one model, bids, or a council?"
```

For local development:

```bash
python -m pip install -e ".[dev]"
python -m pytest
```

Inspect a route without calling a model:

```bash
llm-market plan "Should this prompt use one model, bids, or a council?"
```

Run locally with deterministic mock models:

```bash
llm-market run --mock --trace "Compare carbon taxes and cap-and-trade."
```

Mock answers are synthetic. Use them to inspect route mechanics, traces, and
cost accounting before spending live provider calls.

The public package and CLI are `llm-market`. The Python import/module is
`llm_market` because Python module names cannot contain hyphens, so
`python -m llm_market` works too.

## Run With Real Models

### Codex CLI (Optional Standalone Provider)

This section is only for running `llm-market run --provider codex` from a
terminal. The in-app Codex skill and model-pinned agents described below do not
require a separately installed Codex CLI.

Codex model ids use `codex:<model-slug>:<reasoning-effort>`, for example
`codex:gpt-5.6-luna:medium`, `codex:gpt-5.6-terra:high`, or
`codex:gpt-5.6-sol:max`. Supported effort values are `low`, `medium`, `high`,
`xhigh`, `max`, and `ultra`; the selected Codex model determines which values
are available. The adapter shells out to an authenticated local `codex exec`.

Plan with the estimated-cost Codex cards:

```bash
llm-market plan \
  --config examples/codex_estimated_cost_models.json \
  "Compute 19 * 23 and give only the answer."
```

Run a live Codex call:

```bash
llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --max-tokens 140 \
  "Compute 19 * 23 and give only the answer."
```

By default, Codex provider runs ignore user-level Codex config so evals stay
reproducible. Pass `--codex-use-user-config` when you want the user's local
Codex settings involved. Pass `--codex-oss` when you intentionally want Codex
OSS or local-provider mode, such as Ollama or LM Studio.

### Automatic Codex Next Actions

Install once:

```bash
llm-market codex-install
llm-market codex-install --check
```

In Codex, open `/hooks`, trust all eight LLM Market entries, and restart. The
collapsed rows are app-owned `Hook 1` labels; expand them to see the distinct
`LLM Market: ...` descriptions. Submit one ordinary prompt, then run
`llm-market codex-doctor --smoke`; it reports `passed_with_warnings` until that
real automatic route has been observed. From then on, a normal user request is
enough. The hook starts a warm local controller in the background,
routes each meaningful lifecycle event, and adds private delegation context
only when another model action is useful.

The default roles are:

- Luna Medium for simple answers and bounded follow-ups;
- Terra High for normal coding, GitHub/PR/CI work, repair, and tests;
- Sol High for important read-only planning, architecture, research, and review;
- Sol Max for repeated-failure rescue and council synthesis.

A council is a read-only Terra plus Sol High panel with Sol Max synthesis. One
mutator remains responsible for workspace changes. Objective validation
overrides learned estimates, automatic work is capped at six direct worker
actions plus one council, and any controller failure leaves native Codex
unchanged.

The bundled v3 checkpoint is a real context-6, horizon-2 StructuredStateJEPA.
Its low-weight synthetic bootstrap contains 13,090 transitions across 240
grouped semantic bases; it is a useful starting policy, not evidence about live
model quality. Packaging is blocked unless the latent predictor and all seven
action-conditioned consequence targets beat their held-out baselines. Run
`python tools/evaluate_next_action_jepa.py` for the fixed role/lifecycle probes.

No prompt or answer text is persisted. The local controller stores only
allowlisted embeddings, hashes, counters, decisions, and outcome signals. Use
`llm-market codex-install --remove-hooks` to remove only LLM Market's hooks, or
`llm-market codex-uninstall` to remove the complete managed Codex integration
while preserving unrelated skills, agents, and hooks. The doctor verifies the
package, installed assets, controller route, and prompt-persistence boundary;
Codex hook trust and live model entitlement remain manual checks.
See [Automatic Codex Next-Action Routing](docs/codex_next_action.md).

### Cursor Agent CLI

Cursor model ids use `cursor:<cursor-model-slug>`, for example
`cursor:gemini-3.1-pro` or `cursor:claude-4.6-opus-high`. The adapter shells
out to an authenticated local `cursor agent --print` run.

Check Cursor readiness:

```bash
cursor agent status
cursor agent models
```

Run live through Cursor's own model surface:

```bash
llm-market run \
  --provider cursor \
  --config examples/cursor_models.json \
  --max-tokens 400 \
  --trace \
  "Review this patch plan and say what is missing."
```

If you omit `--config`, `--provider cursor` uses a built-in starter Cursor model
universe with Composer, Gemini, and Claude/Opus cards, not Cursor GPT/OpenAI
cards. The starter costs are relative routing weights because Cursor does not
expose stable public per-token prices through this CLI path. Replace them with
your own observed costs, latency, and quality data when you have it.
The starter Cursor cards include `request_overhead_tokens: 60000`, because
Cursor Agent runs can report a large hidden prompt/workspace overhead even for
short user prompts. Replace this reserve with your own observed overhead for
your repo and Cursor workspace.

Cursor runs default to `--cursor-mode ask` and pass `--trust` so unattended
skill runs do not stop at workspace-trust prompts. Use `--cursor-mode plan` for
Cursor's planning mode, `--cursor-no-trust` when you want Cursor to enforce its
trust prompt, and `--cursor-sandbox enabled|disabled` when you want to pass
Cursor Agent's sandbox flag explicitly.

### Mixed Local Agents

Use `--provider multi` when one run should mix local providers. The multiplexer
dispatches by model id:

- `codex:*` goes to Codex CLI
- `cursor:*` goes to Cursor Agent CLI
- other OpenRouter-style ids go to OpenRouter

With no config, `--provider multi` uses the built-in Codex + Cursor starter
cards:

```bash
llm-market run \
  --provider multi \
  --force-council \
  --max-tokens 500 \
  --trace \
  "Have a small council review this architecture choice."
```

Pass `--config` when you want a custom mixed universe or want to add OpenRouter
models. OpenRouter calls still need `OPENROUTER_API_KEY`, but the multiplexer
does not require that key unless an OpenRouter-style model is actually called.

### OpenRouter

Set `OPENROUTER_API_KEY`, then use an OpenRouter model-card config:

```bash
export OPENROUTER_API_KEY=...

llm-market run \
  --provider openrouter \
  --config examples/openrouter_models.json \
  --trace \
  "Find the best route for this code-review prompt."
```

## Codex Skill

This repo includes a Codex skill in `skills/llm-market`. Install that folder
as `~/.codex/skills/llm-market` to call LLM Market from any Codex thread:

```bash
python tools/install_integrations.py codex
python tools/install_integrations.py codex --check
```

The in-app skill uses Codex custom agents directly. Do not install, update,
log in to, or invoke a separate `codex` executable just to use this path.

The installer honors `--codex-home` first, then `CODEX_HOME`, then defaults to
`~/.codex`. It deletes
stale installed files that no longer exist in the repo, preserves local
`INSTALL_RECEIPT.txt`, installs eight model-pinned profiles for four models and
automatic lifecycle hooks, verifies the installed copy, and runs a no-spend wrapper smoke.
Restart Codex after a sync so changed skill and hook metadata is picked up.

The skill wrapper resolves LLM Market in this order:

1. an explicit `--repo` path
2. `LLM_MARKET_REPO`
3. the current repo or a parent directory
4. the skill directory or a parent directory
5. an installed Python package for plain CLI calls
6. a cached GitHub checkout when repo assets are requested or `--update-github`
   is used

If no checkout is found, the wrapper can clone the GitHub repo into
`$CODEX_HOME/llm-market/repo` and run from that cached copy:

```bash
python ~/.codex/skills/llm-market/scripts/llm_market_cli.py \
  --update-github \
  plan \
  "Should this prompt use one model, bids, or a council?"
```

Inside Codex, plain skill calls default to a skill-local Codex-only model-card
file, so they do not select OpenRouter or other non-Codex model ids unless you
explicitly pass `--config` or an explicit non-Codex provider. Plain skill
`live-eval` calls also default to `--provider codex`. Use `--config` when you
intentionally want a custom, OpenRouter, or server-side provider universe.

Run the read-only provider doctor before Codex local-provider work:

```bash
python ~/.codex/skills/llm-market/scripts/codex_provider_doctor.py
```

The skill can distribute the package, examples, eval packs, and wrapper from
GitHub. Live calls still need local provider state: Codex installed and
authenticated, Cursor CLI installed and authenticated, or OpenRouter/OpenAI-
compatible API keys. The skill reports local provider readiness; it does not
edit `~/.codex/config.toml`.

Check all local integration surfaces from the repo:

```bash
python tools/install_integrations.py all --check
```

Today this means a Codex skill sync/check, a no-spend Cursor provider plan
smoke using the built-in Cursor starter cards, and an explicit Claude Code
pending notice. Claude models are available through OpenRouter-style cards and
Cursor cards; a Claude Code local-provider adapter is not checked in yet.

The public, self-contained installation path is the PyPI wheel plus
[`docs/LLM_MARKET_INSTALL.md`](docs/LLM_MARKET_INSTALL.md). Source installs and
the wrapper's GitHub-checkout fallback are maintainer/development paths; they
are not prerequisites for automatic Codex routing.

## How Routing Works

For automatic Codex sessions, the default path is sequential:

1. The local MiniLM encoder turns the transient prompt or answer plus recent
   sanitized trajectory into a state embedding.
2. StructuredStateJEPA consumes up to six states and one candidate action,
   predicts the next latent state, and applies learned consequence heads for
   quality, success, progress, rescue, unnecessary continuation, latency, and
   cost.
3. The deterministic controller removes actions that violate authority,
   validation, concurrency, model availability, or compute limits.
4. It chooses the cheaper action among predictions within 0.03 expected quality
   of the best, with stricter lift gates for Sol Max and councils.
5. Hooks privately ask Codex to accept, delegate, run a bounded council, or
   continue natively. The next lifecycle event closes the loop.

The controller chooses one bounded next action, not one model for the entire
task. It makes a fresh choice when the user submits a message and when Codex
reaches a `Stop` boundary. Tool, worker, validation, and compaction events
update the trajectory in between; compaction does not delay or trigger routing.

```mermaid
flowchart TD
    A["User submits a message"] --> B["Encode the message and restore recent task state"]
    B --> C["JEPA predicts consequences for every legal next action"]
    C --> D["Apply authority, validation, concurrency, and compute gates"]
    D --> E{"Choose the next bounded action"}

    E -->|answer_fast| F["Luna Medium answers"]
    E -->|execute_standard| G["Terra High inspects or implements"]
    E -->|plan_important| H["Sol High plans"]
    E -->|review_or_repair| I["Sol High reviews a failure"]
    E -->|rescue_max| J["Sol Max handles repeated failure"]
    E -->|council| K["Read-only council with Sol Max synthesis"]
    E -->|continue_native| L["Parent Codex continues"]
    E -->|accept_current| M["Finish the task"]

    F --> N["Tool, worker, and validation events update state"]
    G --> N
    H --> N
    I --> N
    J --> N
    K --> N
    L --> N

    N --> O["Codex reaches a Stop boundary"]
    O --> C
    P["PreCompact / PostCompact"] -. "Record state only; no route choice" .-> N
    Q["User follow-up"] --> B
```

A conclusive validation pass ends the loop. A first conclusive failure makes a
Sol High review the legal repair action; repeated failure can unlock Sol Max.
When multiple actions have nearly equal predicted quality, the cheaper one
wins.

The JEPA receives state and action descriptors; it does not receive a hand-
written “easy coding” or “hard research” route rule. Deterministic parsing is
used only to preserve the user's answer/read/write authority. Standalone market
`plan` and `run` remain separate provider-experiment tools; they do not control
the automatic Codex lifecycle.

## Configuration

Model cards are editable beliefs about cost, latency, reliability, context, and
capability:

```json
{
  "models": [
    {
      "id": "openai/gpt-latest",
      "provider": "openai",
      "capabilities": {
        "coding": 0.94,
        "reasoning": 0.92,
        "writing": 0.82
      },
      "input_cost_per_mtok": 5.0,
      "output_cost_per_mtok": 15.0,
      "context_tokens": 256000,
      "latency_ms": 2500,
      "reliability": 0.94,
      "tags": ["structured", "tool-use", "generalist"]
    }
  ]
}
```

Evidence rows record what worked on prior prompts:

```json
{"model":"openai/gpt-4.1-mini","dimensions":{"coding":1.0},"score":1.0,"correct":true,"complexity":0.32,"risk":0.12}
```

Exact or strongly similar prompt history counts more than broad task-family
averages. A single model needs enough margin before it can suppress council
routing.

## Python API

```python
import asyncio

from llm_market import HeuristicMarketJudge, MarketEngine, MockModelClient
from llm_market.presets import default_model_cards

client = MockModelClient.from_static(
    {
        "openai/gpt-latest": "Strong coding and structured reasoning answer.",
        "anthropic/claude-opus-latest": "Careful critique and risk analysis.",
        "google/gemini-pro-latest": "Broad research and long-context synthesis.",
    }
)

engine = MarketEngine(
    model_cards=default_model_cards(),
    client=client,
    judge=HeuristicMarketJudge(),
)

result = asyncio.run(engine.run("Compare two approaches to a risky migration."))
print(result.answer)
print(result.trace.route.to_dict())
```

## CLI Reference

`next-action` inspects the automatic Codex controller without exposing raw
conversation text:

```bash
llm-market next-action --input route_state.json --format json
```

`next-action-serve` runs the warm local Unix-socket process used by hooks. The
installer starts it automatically when needed.

`plan` explains the route without model calls:

```bash
llm-market plan --format json \
  --expected-output-tokens 800 \
  --max-expected-cost-usd 0.02 \
  "Write a SQL query for weekly retention."

llm-market plan --provider cursor \
  "Should Cursor handle this prompt with one model or a council?"

llm-market plan --provider multi \
  "Should local Codex and Cursor agents review this architecture together?"
```

Installed-package fallback is intentionally narrower: it can run the CLI and
default in-package model cards, but repo assets such as `examples/`, `outputs/`,
and `skills/` require a source checkout or the cached GitHub checkout. The
real-model commands below assume you are in a source checkout or the skill has
resolved a cached GitHub checkout.

`run` executes a route:

```bash
llm-market run --mock --trace "Should this question use a council?"
llm-market run --mock --market auto --trace "Compare these migration options."
llm-market run --mock --council off --trace "Force single-model routing."
```

Presets set the operating posture without changing model cards:

- `--preset fast`: single-model, no bids, no council; uses evidence and cost
  pressure to prefer cheap earned routes.
- `--preset balanced`: default evidence/cost-aware routing.
- `--preset max`: bid-first and more quality-seeking; allows larger councils,
  one fallback, and lower cost/latency penalties without blindly forcing a
  council.

```bash
llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence ~/.llm-market/evidence.jsonl \
  --preset fast \
  "Summarize this diff and flag likely bugs."
```

`live-eval` calls real providers and compares the routed answer with baselines:

```bash
llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence ~/.llm-market/evidence.jsonl \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode external \
  --baseline-config examples/codex_complex_baselines.json \
  --provider-timeout-seconds 240 \
  --max-tokens 700 \
  --progress \
  --output ~/.llm-market/runs/new_codex_run.json
```

Use `--baseline-mode external --baseline-config ...` when you want a clean
comparison against direct baselines such as xhigh-for-everything and cheap
fixed-single arms without direct-running every selectable candidate. Omit
`--baseline-mode` to keep the default full-baseline behavior.

For objective eval tasks with graders, `live-eval` also has an experimental
validation ladder. It is off by default. When enabled, the routed answer is
graded first; only a failed grade can trigger same-route retry or forced-council
escalation:

```bash
llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode none \
  --validation-ladder same-route-then-council
```

Use `--best-single auto` for product routing. When similar prior outcomes show
that a selectable model repeatedly dominates, the router can use that model
directly.

`evidence-ingest` turns live-eval artifacts into future routing evidence:

```bash
llm-market evidence-ingest \
  --results ~/.llm-market/runs/<new-run>.json \
  --output ~/.llm-market/evidence.jsonl
```

Evidence can include prompt text, run identity, latency, route metadata, and
model outcomes. Redact or keep private any eval artifact that contains customer,
repo, or personal task text before publishing an evidence store.

`route-episodes export` preserves richer scored routing decisions for local
inspection, shadow evaluation, and future calibration or training:

```bash
llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode selectable \
  --output ~/.llm-market/runs/<run-id>.json

llm-market route-episodes export \
  --results ~/.llm-market/runs/<run-id>.json \
  --output ~/.llm-market/route_episodes.jsonl
```

Compact evidence JSONL is the runtime prior for today's router. Route episodes
JSONL is a richer private artifact with prompt profile, candidates, policy,
route action, scored outcome, baselines, and counterfactuals. Route episodes are
local/private by default: LLM Market does not upload, sync, phone home, collect
telemetry, or share them automatically. Use `--redact-prompt` when exporting a
dedupable prompt-hash-only copy. Future global route models should be trained
only from maintainer-controlled, public/shareable, or explicit opt-in artifacts.

The only bundled Codex routing model is
`builtin:codex-next-action-jepa-v3`. Older whole-route, sticky-epoch,
route-world, sufficiency, and Gate 2 artifacts are not included in the wheel.
The generic market CLI still accepts explicit local research advisors by path,
but none participate in automatic Codex routing.

### Offline ability-space research

The standalone research CLI can place a model into an explicit local
ability-space artifact from graded episodes:

```bash
llm-market route-world onboard-model \
  --ability-space runs/my_ability_space.json \
  --model-id my-new-model \
  --episodes runs/my_new_model_episodes.jsonl \
  --output runs/my_new_model_placement.json
```

No ability-space model is bundled or used by automatic Codex routing. See
`docs/model_onboarding.md` for the local-artifact workflow.

## Repo Map

- `src/llm_market/analysis.py`: transparent profiling for standalone market
  experiments.
- `src/llm_market/routing.py`: standalone provider-market route selection.
- `src/llm_market/bidding.py`: model self-bids for market routing.
- `src/llm_market/evidence.py`: prior-result observations and evidence deltas.
- `src/llm_market/route_episodes.py`: richer local route episode export schema.
- `src/llm_market/next_action.py`: automatic Codex state/action contracts and
  policy.
- `src/llm_market/next_action_server.py`: local lifecycle controller.
- `src/llm_market/route_world.py`: explicit local research-advisor runtime.
- `src/llm_market/sufficiency.py`: explicit local sufficiency experiments.
- `src/llm_market/engine.py`: route execution and fallback behavior.
- `src/llm_market/judge.py`: heuristic judge and LLM judge.
- `src/llm_market/clients.py`: OpenRouter/OpenAI-compatible client and Codex
  CLI client.
- `examples/codex_estimated_cost_models.json`: cost-aware Codex model cards.
- `examples/codex_complex_multistep_tasks.jsonl`: harder rubric-graded Codex
  eval tasks.
- `examples/codex_complex_baselines.json`: direct xhigh and cheap-single
  baselines for complex-task evals.
- `skills/llm-market`: Codex skill wrapper and integration notes.
- `tools/install_integrations.py`: sync/check local integration surfaces.

## OpenRouter Fusion Relationship

OpenRouter Fusion made the council primitive legible: ask multiple models, have
a judge compare answers, then produce one response. LLM Market keeps that
primitive open and adds the route policy around it: when to pay for a council,
when to ask for bids, when a cheap model is enough, when prior evidence should
promote one model, and what trace data should be saved for the next prompt.

## Scope

- Codex cost comparisons use estimated relative costs because Codex CLI does not
  expose per-call billing.
- Provider-specific cache pricing, KV-cache accounting, and cache-hit discounts
  are future cost-model work.
- The included live result is a small complex-task eval. Run your own task
  families before trusting any routing policy in production.
- Model cards need current prices, reliability estimates, and observed evidence
  to stay useful.

## Third-Party Notices

See `THIRD_PARTY_NOTICES.md`.
