Metadata-Version: 2.4
Name: agentspec
Version: 0.3.1
Summary: Turn website traffic into OpenAPI specs and MCP servers for AI agents
Project-URL: Homepage, https://github.com/huwhitememes/agentspec
Project-URL: Repository, https://github.com/huwhitememes/agentspec
Project-URL: Issues, https://github.com/huwhitememes/agentspec/issues
Project-URL: Changelog, https://github.com/huwhitememes/agentspec/blob/master/CHANGELOG.md
Author: huwhitememes
License-Expression: MIT
License-File: LICENSE
License-File: NOTICES.txt
Keywords: api,api-discovery,cdp,chrome-devtools-protocol,code-generation,mcp,model-context-protocol,openapi,openapi-spec,specification,traffic-capture
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.10
Requires-Dist: phonenumbers>=8.13
Requires-Dist: pydantic>=2.0
Requires-Dist: regex>=2024.0
Requires-Dist: tldextract>=5.0
Requires-Dist: typer>=0.9
Provides-Extra: all
Requires-Dist: graphql-core<4,>=3.2; extra == 'all'
Requires-Dist: httpx>=0.25; extra == 'all'
Requires-Dist: mcp<2,>=0.9; extra == 'all'
Requires-Dist: patchright>=1.60; extra == 'all'
Requires-Dist: playwright>=1.40; extra == 'all'
Requires-Dist: pyyaml>=6.0; extra == 'all'
Requires-Dist: schemathesis<5,>=4.22; extra == 'all'
Provides-Extra: audit
Requires-Dist: httpx>=0.25; extra == 'audit'
Provides-Extra: capture
Requires-Dist: playwright>=1.40; extra == 'capture'
Provides-Extra: capture-stealth
Requires-Dist: patchright>=1.60; extra == 'capture-stealth'
Provides-Extra: cli
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Provides-Extra: generate
Requires-Dist: httpx>=0.25; extra == 'generate'
Requires-Dist: mcp<2,>=0.9; extra == 'generate'
Provides-Extra: graphql
Requires-Dist: graphql-core<4,>=3.2; extra == 'graphql'
Provides-Extra: probe
Requires-Dist: httpx>=0.25; extra == 'probe'
Provides-Extra: test
Requires-Dist: schemathesis<5,>=4.22; extra == 'test'
Provides-Extra: yaml
Requires-Dist: pyyaml>=6.0; extra == 'yaml'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/huwhitememes/agentspec/master/.github/assets/banner.png" alt="agentspec" width="100%">
</p>

# agentspec

Capture website HTTP traffic, discover API endpoints, and produce OpenAPI 3.1 specs + MCP servers for AI agents.

Fully deterministic. No LLM calls. No heuristics requiring judgment.

## Install

```bash
pip install -e .             # core CLI + spec generation (typer is bundled - no extra needed)
# or, for development:
pip install -e ".[dev]"
```

Optional extras:

```bash
pip install -e ".[capture]"           # Playwright browser capture
pip install -e ".[capture-stealth]"   # Stealth Chromium (Patchright) - for Google-SSO / anti-bot sites (see `capture --stealth`)
pip install -e ".[generate]"  # MCP server generation deps (adds the `agentspec-mcp` command)
pip install -e ".[yaml]"      # YAML output
pip install -e ".[test]"      # `agentspec test` (schemathesis-backed API testing)
pip install -e ".[graphql]"   # GraphQL SDL inference (graphql-core)
pip install -e ".[all]"       # Everything
```


## Usage

### Discover from a CDP trace

```bash
agentspec discover --trace ./trace/ --out ./output/
```

### Capture + discover in one step

```bash
agentspec capture --url https://example.com --out ./output/
```

**Sites that block automated browsers** (Google sign-in's "This browser or app
may not be secure", anti-bot firewalls): add `--stealth` to swap in Patchright
(a drop-in stealth Chromium). Needs the optional extra:

```bash
pip install -e ".[capture-stealth]" && patchright install chromium
agentspec capture --url https://app.example.com --stealth --out ./output/
```

**API-key-authenticated APIs (Supabase, Firebase, etc.):** capture a *client* of
the API, not the vendor's admin dashboard. The dashboard authenticates with your
login session (Bearer) against the *management* API, so the service's `apikey:`
header - which lives on the project's *client* API and is what agentspec detects
- is never sent by the browser. agentspec can only see auth the browser actually
sends, so drive a real client app, the in-product API explorer, or a `fetch` that
sets the `apikey` header, and capture *that*. The easiest way: export the
session as a HAR file and use `capture --har` (below).

### Capture from a HAR file (Postman, Insomnia, mitmproxy, curl)

Browser captures can't see API-key headers - browsers never send `apikey:`/
`X-API-Key:` (see the note above). A HAR 1.2 file exported from Postman,
Insomnia, Bruno, Charles, Fiddler, mitmproxy, or Chrome/Firefox DevTools
carries the FULL request headers the client actually sent. Translate it directly
into agentspec's trace format and run the same discover pipeline:

```bash
agentspec capture --har ./traffic.har --out ./output/
```

No optional extra needed (stdlib JSON only). Credentials are redacted on ingest
- they never reach disk. `--probe-auth` works against the live origin afterward
(HARs carry real URLs).

**Multi-origin HARs** (a real client session hits your API + CDNs + analytics):
filter with the same flags `discover` takes:

```bash
agentspec capture --har ./traffic.har --out ./output/ \
  --origin https://api.example.com --exclude "cdn\.|analytics\."
```

**Stateful validation** (`agentspec test --stateful`): a CRUD-shaped HAR
(POST/GET/PATCH/DELETE on `/items/{id}`) produces a linkable spec - browser
captures lack REST CRUD chains and hit `NoLinksFound`. Export a real client
session as HAR, run `capture --har`, then `agentspec test --stateful` against
the emitted spec. A captured HAR is thinner than a hand-authored spec, so
`--stateful` will surface spec-completeness findings (undeclared 4xx responses;
schema-violating requests a lenient server accepts) - point it at a
schema-enforcing target, or treat the findings as useful spec-gap signal.

### Generate an MCP server

```bash
agentspec generate mcp --spec output/openapi.json --out mcp_server.py
```

### Generate with auth headers

Auth headers are wired by **name only** - the generated server reads each secret from
an `AGENTSPEC_AUTH_<NAME>` environment variable at runtime, so secrets are never written
into the generated source or passed on the `agentspec` command line:

```bash
agentspec generate mcp --spec output/openapi.json --auth-header Authorization --out mcp_server.py
```

At server runtime, set the env var (header name upper-cased, non-alphanumerics → `_`).
To keep the secret out of shell history, use `read -s` or a secret manager rather than
a literal `export '...'`:

```bash
read -rs AGENTSPEC_AUTH_AUTHORIZATION   # type the bearer token (no echo), press Enter
export AGENTSPEC_AUTH_AUTHORIZATION
python mcp_server.py
```

### Test an API against its spec

`agentspec test` generates [schemathesis](https://github.com/schemathesis/schemathesis)
property-based tests from an emitted spec and runs them against the live API.
It needs the optional `[test]` extra (schemathesis pulls a heavy transitive graph -
hypothesis, jsonschema-rs, pytest>=9 - which is why it is opt-in):

```bash
pip install -e ".[test]"
agentspec test --spec output/openapi.json
```

The target server defaults to the spec's `servers[0].url`; override with `--origin`.
Auth uses the **same name-only convention** as `generate mcp` - secrets come from
`AGENTSPEC_AUTH_<NAME>` env vars at test-collection time. Extra args after `--` go to
pytest:

```bash
agentspec test --spec output/openapi.json \
  --origin https://staging.example.com \
  --auth-header Authorization \
  -- -x --maxfail=5
```

The exit code is pytest's (`0` = clean, `1` = findings, `2+` = error), so `agentspec test`
drops straight into CI. Pass `--keep-dir PATH` to inspect the generated test module.

#### Stateful testing (`--stateful`)

The default mode runs **stateless** tests - one property-based test per operation.
Add `--stateful` to instead run **stateful** tests that chain operations via
OpenAPI `links` (create → read → update → delete → confirm-gone), surfacing
state-dependent bugs the stateless mode structurally cannot. Schemathesis's
dependency inference infers the links from the spec's shape (path params +
response fields), so a captured CRUD spec needs no hand-written links.

```bash
agentspec test --spec output/openapi.json --stateful
```

**This is destructive.** Stateful testing actively creates / modifies / deletes
resources on the target as it explores state - more dangerous even than
`--coverage`. Only point it at a disposable / staging / **localhost** target you
own. It errors (`NoLinksFound`) on specs with no linkable producer/consumer
operations - e.g. a browser capture with no REST CRUD chains; an API-client
capture (a real client session, not a browser) is the input shape that produces
linkable specs.

### OAuth `securitySchemes` detection

When the captured session includes an OAuth authorization leg (a GET to an
`/oauth*` / `/openid-connect` endpoint carrying `response_type` or `redirect_uri`),
`agentspec discover` emits an OpenAPI `securitySchemes.OAuth2` entry and attaches
a `security` requirement to the operations that actually carried a Bearer token.

Coverage and honesty notes:

- **authorizationCode** legs are recorded under a `flows.x-observed-authorization-code`
  extension. The standard `authorizationCode` flow is intentionally NOT emitted
  because its required `tokenUrl` is not observable from the browser-redirect leg
  (inventing one would be dishonest); the extension documents what was seen. Fill
  `tokenUrl` from `.well-known/oauth-authorization-server` if you need a strict-valid
  standard flow. **implicit** legs (where `tokenUrl` is not required) use the
  standard `flows.implicit` object.
- `security` is attached **per-operation** (only on Bearer-bearing ops), never
  top-level, so public endpoints are not falsely marked OAuth-mandatory. An
  endpoint is marked mandatory only when every successful sample carried Bearer
  (a 401-then-Bearer retry sequence still qualifies).
- `client_id`, `state`, and `redirect_uri` **values** never reach the spec - only
  the `authorizationUrl` (query/matrix-params/userinfo stripped), scope names,
  and grant type. The capture redactor preserves just the auth scheme
  (`Authorization: Bearer abc` → `Bearer <redacted>`) so detection works on real
  captured traces without persisting the credential.
- `--exclude` is honored (an excluded IdP URL contributes nothing); `--origins`
  / `--include` do not filter the scan, because the IdP authorization leg is
  inherently cross-origin from the protected API.

The redirect_uri bypass vulnerability fuzzer deferred from this detection
milestone is now shipped as `agentspec audit oauth` - see the next section.

### Audit OAuth `redirect_uri` bypasses

`agentspec audit oauth` is an ACTIVE vulnerability fuzzer: it takes an OAuth
authorization URL, tampers the `redirect_uri` across a 27-scenario parser-confusion
taxonomy (evil-domain substitution, `@` username tricks, `//` no-scheme,
`%2f`/`%5c`/`%23` domain bypasses, IDN homograph, scheme/IPv6 tricks, ...), fires
each at the IdP's `/authorize` endpoint, and classifies the first-hop response as
**VULNERABLE** / **BLOCKED** / **AMBIGUOUS**. It tests the IdP's redirect_uri
*validator* - the "evil" host never needs to exist.

```bash
pip install -e ".[audit]"   # httpx (reuses [probe])

# from a full authorization URL:
agentspec audit oauth --i-am-authorized \
  --authz-url 'https://idp.example.com/oauth/authorize?response_type=code&client_id=...&redirect_uri=...' \
  --out ./audit-report

# or extract the authorization URL from a captured CDP trace:
agentspec audit oauth --i-am-authorized --from-trace ./trace/ --origin https://idp.example.com --out ./audit-report
```

**Authorization gate (load-bearing):** this is an active attack tool - it sends
crafted bypass attempts at a live OAuth server. `--i-am-authorized` is required
(every run prints the disclaimer); only run it against systems you own or are
authorized to assess. Unauthorized use may violate the law (e.g. CFAA) and the
target's terms. The operator bears responsibility; the tool warns and gets out of
the way - no target restriction beyond the flag.

The verdict is **structural, not prose**: VULNERABLE iff the IdP would hand the
browser to the attacker host (3xx `Location` → attacker, or a 200 body that
auto-navigates / auto-submits a form to it); BLOCKED iff 4xx or an OAuth
`error=redirect_uri_mismatch`; else AMBIGUOUS. The JSON + Markdown report is
credential-redacted at the write boundary (`code`/`state`/`access_token` values
never persist). Validated against real IdPs (Keycloak, Auth0) - both correctly
reject all 26 bypasses (0 VULNERABLE).

## Pipeline

```
CDP Capture → load → filter → normalize → infer → emit
              pair     drop     templatize   schema     openapi.yaml
              req/resp  noise    paths,       from       openapi.json
              bodies    by       GraphQL/     samples    report.md
                        domain   RPC         redact     confidence.json
                                             PII        mcp_server.py
```

5 stages, fully offline. Each stage is a pure function you can call independently:

```python
from agentspec.pipeline.load import load_from_dir
from agentspec.pipeline.filter import filter_pairs
from agentspec.pipeline.normalize import normalize
from agentspec.pipeline.infer import infer_schemas
from agentspec.pipeline.emit import emit
```

## Output

| File | Description |
|---|---|
| `openapi.yaml` | OpenAPI 3.1 spec with component hoisting |
| `openapi.json` | Same spec in JSON |
| `confidence.json` | Per-endpoint confidence scores (low/medium/high) |
| `report.md` | Human-readable summary with curl examples |

## Development

```bash
pytest              # 529 tests (run `pytest tests/` to see the current count)
pytest -v           # verbose
pytest --cov        # coverage
```

## Architecture

- **schema/** - Associative JSON Schema merge, path templating, pattern-based PII redaction (identifiers like email/phone/card/SSN/IBAN; **not** free-text names/addresses - see `redact.py` for the full scope)
- **pipeline/** - 5-stage discover pipeline (load, filter, normalize, infer, emit)
- **generators/** - MCP server code generation
- **capture/** - CDP traffic capture via Playwright
- **cli.py** - Typer CLI

## License

MIT. See [LICENSE](LICENSE) and [ATTRIBUTION.md](ATTRIBUTION.md).
