Metadata-Version: 2.4
Name: tdqs
Version: 0.1.0
Summary: Reference implementation of the Tool Definition Quality Score (TDQS): score how well an MCP tool definition communicates to an AI agent.
Keywords: lint,mcp,model-context-protocol,quality,tdqs,tool-definition
Author: Glama
License-Expression: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Dist: anyio>=4.0
Requires-Dist: exceptiongroup>=1.2 ; python_full_version < '3.11'
Requires-Dist: httpx>=0.28
Requires-Dist: mcp>=2.0,<3
Requires-Dist: pydantic>=2.11
Requires-Dist: typing-extensions>=4.12
Requires-Python: >=3.10
Project-URL: Homepage, https://tdqs.dev
Project-URL: Repository, https://github.com/glama-ai/tdqs
Project-URL: Specification, https://github.com/glama-ai/tool-definition-quality-score
Description-Content-Type: text/markdown

# tdqs

The [Tool Definition Quality Score](https://github.com/glama-ai/tool-definition-quality-score) (TDQS) for Python: a CLI and a library that score how well an MCP tool definition communicates to an AI agent, exactly as the specification defines it. It is the same reference implementation that ships for Node as the `tdqs` npm package, stage for stage, and the two produce the same numbers, the same hashes and the same prompts.

TDQS scores a **definition**, not behaviour. The inputs are what an MCP client sees from `tools/list` — name, title, description, input schema, output schema, annotations — and the output is a score from 1.0 to 5.0 with a letter tier, per tool and per server, with a justification for every dimension.

## Install

```bash
pip install tdqs
# or run it without installing
uvx tdqs --help
```

Python 3.10 or newer.

## Lint: deterministic, no model, no key

```bash
tdqs lint --file tools.json
tdqs lint --command "uvx my-mcp-server"
tdqs lint --url https://mcp.example.com/mcp --header "Authorization: Bearer …"
```

`lint` runs the stages of the pipeline that need no model: the context signals (parameter counts, schema description coverage, annotation values, invocation cost, the definition's hash and byte size), the hard gates (no description, tautological description), the shadow prefilter across the tool set, and the checklist the specification ranks highest. It exits `1` on an error-level finding, which makes it a pull request check:

```bash
tdqs lint --file tools.json --fail-on warning --format markdown --output tdqs-lint.md
```

A lint finding names a fix. It is not a score, and it never pretends to be one.

## Score: the full rubric

```bash
export TDQS_BASE_URL=https://api.openai.com/v1   # any OpenAI-compatible endpoint
export TDQS_API_KEY=…
export TDQS_MODEL=…

tdqs score --file tools.json
tdqs score --command "uvx my-mcp-server" --fail-under B --format markdown
```

`score` sends every tool through the rubric (six dimensions, 1–5 each, with the specification's system prompt verbatim), runs the server coherence evaluation (four dimensions plus shadowing-risk confirmation), and rolls both up into the server score with integer arithmetic. The report is stamped with the specification version and the model, because a score is calibrated to a rubric+model pair and is not comparable to anything without both.

Turn extended reasoning off. The reference model reasons before it answers unless told not to, which makes a call take a minute instead of seconds — and the specification's calibration examples reproduce with reasoning **off**. How to say so is provider-specific, so it is an opaque JSON object merged into every request:

```bash
tdqs score --file tools.json --request-overrides '{"reasoning":{"enabled":false}}'   # OpenRouter
# or TDQS_REQUEST_OVERRIDES in the environment; DeepSeek directly takes {"thinking":{"type":"disabled"}}
```

`--hosted https://tdqs.dev` scores through a hosted TDQS site instead of a model key of your own, and prints the report's URL. It takes that site's API key as `--api-key` or `TDQS_API_KEY`; the site's account page is where keys come from.

Input is exactly one of `--file` (a `tools/list` result, an array of tools, or a single tool; `-` reads stdin), `--command` (a stdio server) or `--url` (a Streamable HTTP server).

| Exit code | Meaning                                                                    |
| --------- | -------------------------------------------------------------------------- |
| `0`       | done                                                                       |
| `1`       | the threshold was not met (`--fail-on` for lint, `--fail-under` for score) |
| `2`       | usage error, unreadable input, unreachable server, or a model failure      |

`--format` is `text` (default), `markdown` or `json`. The JSON formats are the ones the npm package publishes as JSON Schema, `schemas/score-report.json` and `schemas/lint-report.json`.

## Library

```python
from tdqs import create_llm_client, lint_server, parse_tool_definitions, score_server

parsed = parse_tool_definitions(response.json())
server_name = parsed.server_name or "my-server"

# No model involved.
lint = lint_server(server_name=server_name, tools=parsed.tools)

# The full pipeline.
report = score_server(
    llm=create_llm_client(
        api_key=api_key,
        base_url=base_url,
        model=model,
        request_overrides={"reasoning": {"enabled": False}},
    ),
    server_name=server_name,
    tools=parsed.tools,
)

report.server_score.overall_tier  # "A" | "B" | "C" | "D" | "F"
report.tools[0].justifications["usage_guidelines"]  # Justification(score=…, justification=…)
report.model_dump()  # the specification's JSON, camelCase keys
```

Every stage is exported on its own — `compute_context_signals`, `evaluate_hard_gates`, `compute_tdqs`, `find_shadow_candidates`, `build_tool_scoring_prompt`, `score_tool_definition`, `score_server_coherence`, `rollup_server_score` — along with the specification's metadata (`TOOL_DIMENSIONS`, `COHERENCE_DIMENSIONS`, `FLAGS`, `TIERS`, `LINT_RULES`, `SPEC_VERSION`) and the two system prompts, so a registry or a gateway can build on the same pieces. Reports are pydantic models; `model_dump()` is the specification's JSON and `ScoreReport.model_validate()` reads it back.

`request_hosted_report(...)` is the hosted mode as a function: it submits the definitions to a TDQS site and polls until the report is done.

## What is deterministic and what is not

Stages 1, 2 and 4 of the pipeline, the shadow prefilter, and every rollup are deterministic and reproducible from the definitions alone; `inputHash` is computed the same way the Glama registry computes it, so a hash here matches the one on a server's public score page. Stage 3 — the rubric — and the coherence evaluation are model calls. The specification pins the prompts, the output contract and the calibration examples; the model is the remaining variable, which is why every report names it. Swap models and expect to re-score.

## Specification

This package follows TDQS **1.2**. The prompts are compared byte for byte against the specification in the test suite, and the deterministic stages are compared against the Node reference implementation's fixtures, so the implementations cannot drift apart silently.
