Metadata-Version: 2.4
Name: muckrock-mcp
Version: 0.4.3
Summary: Model Context Protocol (MCP) server for MuckRock Requests and DocumentCloud APIs
Author: Phil Mocek
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx<0.29.0,>=0.27.0
Requires-Dist: uvicorn>=0.30.0
Requires-Dist: anyio>=4.0.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
Requires-Dist: pytest-httpx>=0.30.0; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: tomli>=1.2.0; python_version < "3.11" and extra == "dev"
Dynamic: license-file

# MuckRock & DocumentCloud Model Context Protocol (MCP) Server

[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://python.org)
[![MCP Version](https://img.shields.io/badge/MCP-2026--07--28-green.svg)](https://modelcontextprotocol.io)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

A [stateless](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http) [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server providing structured access to [MuckRock](https://www.muckrock.com/api_v2) Requests and [DocumentCloud](https://www.documentcloud.org/help/api/) APIs — eight read-only tools, six resource templates, and dual stdio/HTTP transports (see [Features](#features)).

---

## Overview: LLMs & Public Records (FOIA/PRA)

[MuckRock](https://www.muckrock.com) is an open-government platform that helps journalists, researchers, and anyone seeking to hold public institutions accountable file, track, and share public records requests.  [DocumentCloud](https://documentcloud.org/home/) is an open repository and analysis tool for primary source documents released by government agencies.

This [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server bridges AI agents (such as [Goose](https://github.com/aaif-goose/goose), [OpenCode](https://github.com/OpenCode-AI/opencode), [Hermes Agent](https://github.com/NousResearch/Hermes), [Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview), [Codex](https://openai.com/index/openai-codex/), or custom local LLM pipelines) with [MuckRock](https://www.muckrock.com) and [DocumentCloud](https://www.documentcloud.org) data. Rather than requiring models to parse arbitrary web pages, this server provides structured, machine-readable access to:
- **Investigative Discovery**: Searching dockets, tracking communication logs between requesters and agencies, and inspecting responsive records.
- **Document Analysis**: Searching released document corpuses, fetching clean page-level extracted plain text, and extracting identified entities (people, organizations, locations).
- **Automated Workflows**: Resolving agency identifiers, monitoring docket status changes, and auditing statutory response deadlines.

All interactions adhere to the standardized [Model Context Protocol](https://modelcontextprotocol.io), allowing AI hosts to dynamically discover tools, resolve URI resources directly into model context, and autocomplete request parameters.

---

## Architecture Overview

```
                         ┌──────────────────────────────┐
                         │    LLM Client / MCP Host     │
                         │ (OpenCode, Goose, Codex etc) │
                         └──────────────┬───────────────┘
                                        │
                         ┌──────────────┴───────────────┐
                         │       Transport Layer        │
                         │  StdioServerTransport (CLI)  │
                         │   StreamableHTTPTransport    │
                         └──────────────┬───────────────┘
                                        │
                         ┌──────────────┴───────────────┐
                         │      JSONRPCDispatcher       │
                         │  (MCP 2026-07-28, Discover)  │
                         └──────────────┬───────────────┘
                                        │
         ┌──────────────────────────────┼──────────────────────────────┐
         │                              │                              │
┌────────┴────────┐            ┌────────┴────────┐            ┌────────┴────────┐
│ CatalogManager  │            │ Tools & Registry│            │Resources/Resolve│
│ • Deterministic │            │ • 8 FOIA Tools  │            │ • 6 Templates   │
│   Sorted Lists  │            │ • Structured Out│            │ • RFC 6570 Match│
│ • Cache Direct. │            │ • ResourceLinks │            │ • Autocomplete  │
└─────────────────┘            └────────┬────────┘            └─────────────────┘
                                        │
                               ┌────────┴────────┐
                               │ Upstream Client │
                               │ MuckRockClient  │
                               │ DocumentCloud   │
                               └────────┬────────┘
                                        │
                               ┌────────┴────────┐
                               │   Auth & Rate   │
                               │ TokenStore(JWT) │
                               │ TokenBucket 15/5│
                               │ Jitter Backoff  │
                               └─────────────────┘
```

---

## Features

- **Ingress Transports**:
  - **Standard I/O (`stdio`)**: Fast, newline-delimited JSON-RPC 2.0 streaming over standard input/output with diagnostic logging strictly isolated to standard error.
  - **Streamable HTTP (ASGI 3.0)**: Stateless `POST /mcp` command ingress (per-request JSON responses) with a strict 4MB framing safeguard. `GET`/`DELETE` on the endpoint return `405` — no server-side sessions.
- **Squarelet OAuth / JWT Authentication**: Proactive token refresh when access token TTL drops below 60 seconds (`POST https://accounts.muckrock.com/api/token/refresh/`). Falls back gracefully to unauthenticated mode for public FOIA browsing when no credentials are configured; explicitly configured credentials that the upstream rejects fail loudly with `401` instead of silently serving public-only results.
- **Dual-Tier Sliding-Window Token Bucket**:
  - **General Tier**: 15 req/min (0.25 tokens/s) with burst capacity up to 100 tokens.
  - **Sensitive Tier**: 5 req/min (~0.0833 tokens/s) with zero burst capacity (capacity 1) for agency, organization, and user endpoints.
  - **Jittered Exponential Backoff**: Automatically parses `Retry-After` (seconds or RFC 7231 / RFC 2822 HTTP-date) upon HTTP 429 and applies full jitter backoff.
- **Deterministic Prompt Cache Optimization**: All catalog listings (`tools/list`, `resources/list`, `resources/templates/list`) are lexicographically sorted to maximize LLM prompt cache hit rates, returning `ttlMs: 300000` (5 min) and `cacheScope: "public"`.
- **Eight Core Tools** ([catalog below](#tool-catalog)): Comprehensive FOIA search, request dockets, chronological correspondence threads, agency lookups, DocumentCloud search, full-text extracted plain-text retrieval, and categorized entity extraction. Tool responses embed typed `ResourceLink` objects connecting returned data to matching [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570) URI templates.
- **Six RFC 6570 Resource Templates** ([resolvers below](#resource-templates)): Concrete URI resolvers for requests, threads, agencies, documents, pages, and entities.
- **Argument Autocompletion** ([details below](#autocompletion-completioncomplete)): Fast MCP autocompletion (`completion/complete`) for request statuses, jurisdictions, and dynamic agency search.
- **Opt-in Ingress Authentication**: Set `MCP_API_TOKEN` to require `Authorization: Bearer <token>` on HTTP ingress (unset means open); unauthenticated `GET /healthz` reports liveness. CORS is loopback-only by default — no wildcard.

---

## Protocol Support Statement (Dual-Era Contract)

Requests without the `MCP-Protocol-Version` header are served as protocol version 2025-03-26 (legacy headerless clients are tolerated as 2025-03-26 era, spec-permitted, no elevated privilege). `server/discover` works with zero prior `initialize` — no handshake or session is required. `subscriptions/listen` and live `list_changed` notifications are not supported: the catalog is static with `listChanged: false`, so poll `server/discover` instead.

### Era-by-Transport Matrix

| Era / transport row | Status | Pinning test |
|---|---|---|
| HTTP modern headered (match served) | Supported | `tests/integration/test_http_server.py::TestProtocolVersionHeader::test_match_served_normally` |
| HTTP header/body mismatch → `-32020` | Supported | `tests/integration/test_http_server.py::TestProtocolVersionHeader::test_mismatch_yields_32020` |
| HTTP unsupported version → `-32022` + supported payload | Supported | `tests/integration/test_http_server.py::TestProtocolVersionHeader::test_unsupported_yields_32022` |
| HTTP legacy headerless `initialize` tolerated as 2025-03-26 | Supported | `tests/integration/test_http_server.py::TestProtocolVersionHeader::test_legacy_headerless_served` |
| HTTP legacy headerless `server/discover` tolerated as 2025-03-26 | Supported | `tests/integration/test_http_server.py::TestProtocolVersionHeader::test_legacy_headerless_discover_served` |
| HTTP legacy headerless `tools/list` tolerated as 2025-03-26 | Supported | `tests/integration/test_http_server.py::TestProtocolVersionHeader::test_legacy_headerless_tools_list_served` |
| HTTP stateless posture (no sessions, `405` on `GET`/`DELETE`, forged IDs ignored) | Supported | `tests/integration/test_http_server.py::TestHTTPIntegration::test_full_http_stateless_workflow` |
| Zero-handshake `server/discover` (no prior `initialize`) | Supported | `tests/unit/test_protocol.py::TestJSONRPCDispatcher::test_no_handshake_calls_succeed` |
| `listChanged: false` static catalog | Supported | `tests/unit/test_protocol.py::TestJSONRPCDispatcher::test_no_handshake_calls_succeed` |
| `subscriptions/listen` → `-32601` Method not found | Not supported (by design) | `tests/unit/test_protocol.py::TestJSONRPCDispatcher::test_subscriptions_listen_method_not_found` |
| stdio legacy conversation (`initialize` → `tools/call` → `resources/read` → `completion`) | Supported | `tests/integration/test_stdio_server.py::TestStdioIntegration::test_full_stdio_conversation` |
| Goose harness | Unverified as of 2026-09-08 | — |
| OpenCode harness | Unverified as of 2026-09-08 | — |
| Hermes Agent harness | Unverified as of 2026-09-08 | — |
| Claude Code harness | Unverified as of 2026-09-08 | — |
| Codex harness | Unverified as of 2026-09-08 | — |

---

## Installation

### For users

#### From PyPI

Run the server without touching source code, in an isolated environment:

```bash
# From PyPI, using uv (recommended)
uv tool install muckrock-mcp

# From PyPi, using pip
pip install muckrock-mcp

# With pipx
pipx install git+https://github.com/pmocek/muckrock-mcp
```

#### From a wheel

A [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/)
(`.whl`) is Python's installable package format — handy for air-gapped
machines or pinning an exact build. Build one from a checkout, copy the
single `.whl` file anywhere, and install it:

```bash
# In the checkout
uv build   # writes dist/muckrock_mcp-*.whl

# On the target machine (uv or pipx)
uv tool install muckrock_mcp-0.1.0-py3-none-any.whl
# or
pipx install muckrock_mcp-0.1.0-py3-none-any.whl
```

### For contributors

Install with dev tools to run the test suite (`uv run pytest tests/`)
or contribute code (formatting via `black`). The dev extras are
[`pytest`, `pytest-asyncio`, `pytest-httpx`, and `black`](pyproject.toml).

```bash
# Standard checkout install
uv sync

# With test and development dependencies
uv sync --extra dev
```

Without uv:

```bash
pip install -e .
pip install -e ".[dev]"
```

---

## Configuration & Environment Variables

Set environment variables `MUCKROCK_USERNAME` / `MUCKROCK_PASSWORD` with
valid credentials; without them the server cannot authenticate to MuckRock,
and provides very limited utility (mainly to verify operation).
For HTTP mode, also set `MCP_API_TOKEN` to require a bearer token on ingress.

| Variable | Description | Default |
|---|---|---|
| `MUCKROCK_USERNAME` | MuckRock / Squarelet account username | `None` (Anonymous mode) |
| `MUCKROCK_PASSWORD` | MuckRock / Squarelet account password | `None` (Anonymous mode) |
| `MCP_TRANSPORT` | Server transport mechanism (`stdio` or `http`) | `stdio` |
| `MCP_HOST` | Host interface to bind HTTP transport | `127.0.0.1` |
| `MCP_PORT` / `PORT` | Port number to bind HTTP transport | `8000` |
| `MCP_API_TOKEN` | Bearer token required on HTTP ingress (`Authorization: Bearer <token>`); unset means open | `None` (open) |
| `MCP_DEBUG` | Enable verbose diagnostic logging (`1`, `true`, `on`) | `false` |
| `MCP_CACHE_TTL_MS` | Prompt cache TTL hint in milliseconds | `300000` (5 minutes) |
| `MCP_CACHE_SCOPE` | Prompt cache scope hint (`public`/`private`) | `public` |
| `MCP_MAX_MESSAGE_SIZE` | Max ingress JSON-RPC bytes | `4194304` (4MB) |
| `MCP_CORS_ORIGINS` | Comma-separated CORS allowlist; default 127.0.0.1-only (no `*`) | `` (loopback only) |

---

## Running the Server

### Command-Line Interface (`muckrock-mcp`)

The server provides a unified CLI with full argument parsing:

```bash
# Run over stdio (default)
muckrock-mcp

# Run over Streamable HTTP on custom port
muckrock-mcp --transport http --host 0.0.0.0 --port 8080

# Authenticate via env vars
MUCKROCK_USERNAME="my_user" MUCKROCK_PASSWORD="my_secret" muckrock-mcp

# Enable verbose debug logging
muckrock-mcp --debug

# Show version
muckrock-mcp --version
```

### Health Check

When running over HTTP, liveness (upstream reachability + rate-limiter state)
is exposed without authentication:

```bash
# GET /healthz -> {"status": "ok", ...}
curl http://127.0.0.1:8000/healthz
```

### Direct Python Invocation
```bash
python -m muckrock_mcp.cli --transport stdio
```

---

## Client Integration

### OpenCode
Add to your OpenCode configuration file (`opencode.json`):

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "muckrock": {
      "type": "local",
      "command": ["muckrock-mcp", "--transport", "stdio"],
      "environment": {
        "MUCKROCK_USERNAME": "your_muckrock_username",
        "MUCKROCK_PASSWORD": "your_muckrock_password"
      }
    }
  }
}
```

### Goose CLI
```bash
goose configure
# Select command-line extension and enter:
muckrock-mcp --transport stdio
```

Or add a stdio command extension directly to `~/.config/goose/config.yaml`:

```yaml
# ~/.config/goose/config.yaml
extensions:
  muckrock:
    enabled: true
    type: stdio
    # stdio command extension: launches muckrock-mcp with stdio transport
    cmd: muckrock-mcp
    args:
      - --transport
      - stdio
    envs:
      MUCKROCK_USERNAME: your_muckrock_username
      MUCKROCK_PASSWORD: your_muckrock_password
```

### Hermes Agent
Add to your Hermes Agent configuration file (`~/.hermes/config.yaml`):

```yaml
# ~/.hermes/config.yaml
mcp_servers:
  muckrock:
    command: muckrock-mcp
    args:
      - --transport
      - stdio
    env:
      MUCKROCK_USERNAME: your_muckrock_username
      MUCKROCK_PASSWORD: your_muckrock_password
```

### Claude Code
Add via the Claude Code CLI:

```bash
claude mcp add muckrock \
  --env MUCKROCK_USERNAME=your_muckrock_username \
  --env MUCKROCK_PASSWORD=your_muckrock_password \
  -- muckrock-mcp --transport stdio
```

Or add to your project's `.mcp.json`:

```json
{
  "mcpServers": {
    "muckrock": {
      "command": "muckrock-mcp",
      "args": ["--transport", "stdio"],
      "env": {
        "MUCKROCK_USERNAME": "your_muckrock_username",
        "MUCKROCK_PASSWORD": "your_muckrock_password"
      }
    }
  }
}
```

### Claude Desktop
Add to your Claude Desktop configuration file (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "muckrock": {
      "command": "muckrock-mcp",
      "args": ["--transport", "stdio"],
      "env": {
        "MUCKROCK_USERNAME": "your_muckrock_username",
        "MUCKROCK_PASSWORD": "your_muckrock_password"
      }
    }
  }
}
```

### Codex
Add to your Codex configuration file (`~/.codex/config.toml`):

```toml
[mcp_servers.muckrock]
command = "muckrock-mcp"
args = ["--transport", "stdio"]

[mcp_servers.muckrock.env]
MUCKROCK_USERNAME = "your_muckrock_username"
MUCKROCK_PASSWORD = "your_muckrock_password"
```

---

## Tool Catalog

The server exposes eight idempotent, read-only tools:

| Tool Name | Parameters | Description |
|---|---|---|
| `search_requests` | `query` (str, optional), `agency` (int/str, optional), `jurisdiction` (int/str, optional), `status` (str, optional), `has_files` (bool, optional), `page` (int, default 1) | Search public records requests across MuckRock by keyword, agency, jurisdiction, or status. |
| `get_request_docket` | `request_id` (int, required) | Retrieve top-level docket data, filing dates, jurisdiction, and status for a FOIA request. |
| `get_request_thread` | `request_id` (int, required) | Retrieve chronological correspondence, communications, sender directions, and attached files. |
| `search_agencies` | `query` (str, optional), `jurisdiction` (int/str, optional), `page` (int, default 1) | Locate government agencies, unique identifiers, and jurisdictional mappings. |
| `get_agency_profile` | `agency_id` (int, required) | Retrieve agency details, contact email, portal existence, and historical disposition statistics. |
| `search_documents` | `query` (str, required), `page` (int, default 1), `per_page` (int, default 10) | Query public documents on DocumentCloud matching keywords or queries. |
| `get_document_text` | `document_id` (str/int, required), `page_number` (int, optional), `max_chars` (int, default 20000) | Retrieve full or page-specific extracted plain text from DocumentCloud, capped with truncation hints. |
| `get_document_entities`| `document_id` (str/int, required) | Retrieve categorized entities (people, organizations, locations) identified in DocumentCloud. |

---

## Resource Templates

The server provides six RFC 6570 Resource Templates with URI resolvers:

| URI Template | MIME Type | Resolver Description |
|---|---|---|
| `muckrock://requests/{request_id}` | `application/json` | Request metadata, dates, status, and agency info. |
| `muckrock://requests/{request_id}/thread` | `application/json` | Complete communication thread and correspondence history. |
| `muckrock://agencies/{agency_id}` | `application/json` | Agency details, contacts, and statistical disposition metrics. |
| `doccloud://documents/{document_id}` | `application/json` | DocumentCloud document metadata, title, pages, and source. |
| `doccloud://documents/{document_id}/pages/{page_number}/text` | `text/plain` | Extracted plain text of an individual document page. |
| `doccloud://documents/{document_id}/entities` | `application/json` | Extracted entities (people, organizations, locations). |

---

## Autocompletion (`completion/complete`)

The server implements dynamic argument autocompletion for:
- `status`: Matches against valid MuckRock lifecycle states (`submitted`, `processing`, `done`, `rejected`, `appealed`, `no_docs`, etc.).
- `jurisdiction`: Matches against standard Federal, State, and Territorial jurisdictions.
- `agency`: Queries MuckRock dynamically and formats results as `"{name} (ID: {id})"`.

---

## Testing & Verification

The test harness provides comprehensive, 100% offline verification across unit, integration, end-to-end tiers, and challenger suites:

```bash
# Run the complete test suite
uv run pytest tests/

# Run specific tiers
uv run pytest tests/unit/              # Unit tests (protocol, auth, ratelimit, tools, etc.)
uv run pytest tests/integration/       # Multi-turn transport integration tests
uv run pytest tests/e2e/               # End-to-end multi-tier test suites
uv run pytest tests/e2e/test_tier5_adversarial.py  # Tier 5 Adversarial hardening tests
```

### Test Suite Structure

| Tier | Focus |
|---|---|
| **Unit** | Serialization, models, transports, auth TTL, sliding buckets, tools, lifecycle, auth edges |
| **Integration** | Stdio & Streamable HTTP multi-turn protocol conversations |
| **Tier 1 (Features)** | Happy-path nominal coverage (>=5 tests per feature F01–F22) |
| **Tier 2 (Boundaries)**| Edge cases, limits, and boundary error conditions |
| **Tier 3 (Combinations)**| Pairwise cross-subsystem interactions |
| **Tier 4 (Scenarios)** | 6 Real-world investigative user journeys |
| **Tier 5 (Adversarial)** | Malformed payloads, 4MB overflow, session forgery, path escape, egress caps, burst recovery |
| **Challengers** | Stress, tools/resources cross-checks |
