Metadata-Version: 2.5
Name: digline
Version: 0.15.0
Summary: Regression testing for LLM output, against a baseline committed in your own repo.
Project-URL: Homepage, https://digline.dev/
Project-URL: Documentation, https://digline.dev/product/guide/
Project-URL: Changelog, https://digline.dev/product/changelog/
Project-URL: Repository, https://github.com/digline/digline
Project-URL: Issues, https://github.com/digline/digline/issues
Author-email: Alessandro Prandini <hello@digline.dev>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: assertions,baseline,eval,evaluation,llm,regression,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: jsonschema>=4.21
Description-Content-Type: text/markdown

# digline

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/flow/digline-flow.svg">
  <img src="docs/assets/flow/digline-flow-light.svg" alt="digline in an agentic flow" width="960">
</picture>

---

<br>

**Regression testing for LLM applications — with the baseline in your repository, not on someone's server.**

[![PyPI](https://img.shields.io/pypi/v/digline)](https://pypi.org/project/digline/)
[![Downloads](https://static.pepy.tech/badge/digline/month)](https://pypistats.org/packages/digline)
[![Python 3.12+](https://img.shields.io/pypi/pyversions/digline)](pyproject.toml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-green)](LICENSE)
[![CI](https://github.com/digline/digline/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/digline/digline/actions/workflows/ci.yml)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/14570/badge)](https://www.bestpractices.dev/projects/14570)

Your prompt worked on Tuesday. On Thursday it works a little less — not enough
to break, enough for a user to notice in two weeks. No ordinary test catches it,
because there is no correct output to compare against, only a better or a worse
one.

digline gives you an **approved reference** — the baseline — and on every change
tells you whether you are below it: which case, which check, by how much. The
baseline is a JSON file in your repository, so it goes through code review and
it rolls back with `git`. No server, no account, no network call you have not
configured yourself.

```console
$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).
```

## Why digline

Most evaluation tools tell you whether an output is below a threshold. digline
also tells you whether it is *worse than it was* — the drift from 0.91 to 0.78
that trips no threshold and is the first thing a user feels.

The suite is **Python, not YAML**: a judge is an object, a target is a function,
and what may leave a perimeter is declared in code — none of which a
configuration file expresses without reinventing a language. Where a suite *is*
plain data it can be TOML instead, and the two forms build the same objects —
what TOML cannot express, it refuses by name rather than half-supporting. Built
for teams shipping LLM features for someone else, who have to show a customer
what was tested, when, under which commit, and who approved it.

Wondering how digline differs from promptfoo, DeepEval, or observability
platforms? See [How digline compares](https://digline.dev/comparison/).

## Quickstart

With uv (recommended):

```bash
uv init && uv add digline
```

or with pip in an existing environment: `pip install digline`.

**Requires Python 3.12+**, which uv fetches for you if you do not have it. On
the pip path an older interpreter says `ERROR: No matching distribution found`
with `from versions: none`, which does not say why — that is what it means.

`suite.py` — complete and runnable, no API key:

```python
"""suite.py — complete and runnable: no API key, nothing else to install."""

from digline.core import Contains, CostBudget, JudgeReply, LlmRubric
from digline.run import Case, Response, Suite

ANSWERS = {
    "where-is-my-order": "Order 4821 ships Thursday. — Northwind Support",
    "how-do-i-return": "Any item, within 30 days, unused. — Northwind Support",
}


def judge(prompt: str) -> JudgeReply:
    """Your judge. digline composes `prompt` from the rubric, the question and
    the answer; it wants a score in [0, 1] and a reason back."""
    signed = "Northwind Support" in prompt
    concise = len(prompt.split()) <= 60
    return JudgeReply(
        score=0.4 + 0.3 * signed + 0.3 * concise,
        reason=f"signed={signed}, concise={concise}",
    )


def target(case: Case) -> Response:
    """Your application, called once per case. Canned here so this runs as is."""
    text = ANSWERS[case.id]
    return Response(output=text, cost_usd=0.004 + 0.001 * len(text) / 100)


suite = Suite(
    tenant="northwind",
    environment="staging",
    name="support",
    assertions=[
        Contains(needle="Northwind Support"),
        LlmRubric(
            rubric="Does the reply answer the question in at most three sentences?",
            judge=judge,
            threshold=0.7,
            tolerance=0.05,
        ),
        CostBudget(max_usd=0.02, tolerance=0.05),
    ],
    cases=[Case(id="where-is-my-order"), Case(id="how-do-i-return")],
)
```

When your judge is a real model, add a provider plugin:
`uv add digline-anthropic` (or `pip install digline-anthropic`), likewise
`digline-openai` and `digline-bedrock`.

```console
$ digline run --suite suite.py
2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8

$ digline promote --suite suite.py --run latest
support baseline set to 2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8
```

Now make it worse — delete `— Northwind Support` from the second answer —
and ask again:

```console
$ digline run --suite suite.py
2026-08-26T15-44-09-492722-00-00-e7421ec503ccefe8

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

$ echo $?
1
```

The exit code is the answer: `0` fine, `1` got worse, `2` could not be judged.
Everything lands in `.digline/<tenant>/` — `baselines/` committed, `runs/`
git-ignored through a `.gitignore` digline writes for you.

## What it checks

**Per case** — pure functions `(inputs) -> Verdict`, no I/O, callable on their own:

| Assertion | Use it when |
|---|---|
| `Equals`, `Contains`, `NotContains`, `Affix`, `Regex` | the output must, or must not, contain something specific |
| `IsJson`, `JsonSchema` | the output is structured |
| `Length` | answers are growing, or must fit a channel |
| `Levenshtein` | "close enough" to `Case.expected`, graded rather than binary |
| `LlmRubric` | the criterion is a judgement — is it polite, does it stay on policy |
| `Faithfulness` | RAG: is the answer supported by the retrieved context |
| `FromAutoevals` | you already have an `autoevals` scorer and want it under a baseline |
| `PiiAbsent` | the output reaches a person — IBAN, codice fiscale, partita IVA, email, phone, checksum-verified where one exists |
| `ToolsCalled` | the target is an agent — a function, or an OpenAI, Anthropic or Bedrock target, whose plugins record every tool call — which tools it called, in order: an answer produced without the lookup that should have produced it |
| `ToolCalledWith` | the other half of a trajectory: the arguments a tool was called with — the right tool asked the wrong question |
| `CostBudget`, `LatencyBudget` | always. Graded, so a cost creeping up *within* budget is still visible |
| `Repeated` | the judge oscillates: grade the same output `n` times and fold the votes |

A suite pointed at an OpenAI-compatible endpoint, a gateway or a self-hosted
model declares what it actually charges — `[target.pricing]` in TOML,
`override()` in Python — so a `CostBudget` reads your prices rather than the
plugin's list.

**Per run** — one verdict on the whole suite, the kind that goes in a contract:

| Aggregate | Use it when |
|---|---|
| `Precision` | false positives are what your users see |
| `Recall` | what is missed is what your users miss |
| `Accuracy`, `F1` | you need a single number for both |

Every assertion carries a **threshold that can fail** — there is no default that
passes vacuously, and `Contains("")` is a `ValueError` when the suite loads
rather than a green run — and a **tolerance** below which a difference from the
baseline is noise. Where a number is really "k out of n", write it as one:
`min_agreement="2/3"`, and a float no `k/n` can produce is refused at
construction.

One card each — parameters, typical values, what to watch out for — in
[`docs/metrics.md`](docs/metrics.md). Custom assertion? Subclass
`AssertionBase`, or `RunAssertionBase` for an aggregate: [`docs/api.md`](docs/api.md).

## How it thinks

- **The judge is yours.** The core never calls a model API: you inject a
  function or a provider plugin's judge, and in your tests a deterministic one.
- **Three states, not two** — `pass`, `fail`, `error`. An error is neither green
  nor a regression: it means *could not judge*, and a run containing one cannot
  become the baseline.
- **Two kinds of noise, two answers.** `Suite.samples` asks the target more than
  once — the same input answered differently. `Repeated` grades the same output
  more than once — the judge changing its mind. `min_agreement` becomes
  mandatory as soon as you sample.
- **A tolerance is declared; a noise floor is measured.** A sampled run records
  the interval its own samples spanned, and a drop that stays inside the
  baseline's interval is reported as unchanged rather than as a regression — a
  tool that cries wolf on its own measurement error teaches people to promote
  past it. It never rescues a flip, and it never invents an interval it does not
  have.
- **An alias is a pointer, and pointers roll.** `compare` names the model the
  provider said answered when it changes, `digline log` reads that down every
  stored run, and a `Case(canary=True)` watches behaviour where the provider
  says nothing: if it moves at all, the headline says the model under the alias
  likely changed and the run exits `1`.
- **Set the threshold where the system measurably is**, not where you want it:
  the gate protects against getting worse, and raising the bar is a visible
  change in a pull request.
- **Promote the median of several runs**, not the first green one — `digline
  view` is the table you pick it from. Cases diagnose, aggregates gate.

Worked through with real numbers in [`docs/guide.md`](docs/guide.md); the
reasoning behind every fixed decision is in [`docs/adr/`](docs/adr/).

## Commands

| Command | |
|---|---|
| `digline run` | execute the suite, write the run, print its key. A run killed part way through is finished with `--resume`, which re-pays for the cases nobody has an answer to and nothing else — [`docs/api.md`](docs/api.md#the-run-that-was-killed) |
| `digline compare` | headline plus the lines that got worse; `--json`, `--json full` for CI |
| `digline diff` | what differs between two runs, neither of them a baseline — for "should I switch?" rather than "did it get worse?". Always exits 0: it is a report, not a verdict — [`docs/diff.md`](docs/diff.md) |
| `digline promote` | make a run the baseline — refused if the tenant differs, the configuration changed, or any check errored |
| `digline register` | record what a person decided about a comparison — `--disposition accepted`, `rejected` or `unsure` — as one line under `.digline/<tenant>/register/`, committed like a baseline. Counts and keys only: the reason goes in the message of the commit that adds the line. A person's act, never an agent's or a schedule's — [`docs/register.md`](docs/register.md) |
| `digline report` | self-contained HTML for readers who do not read code; `--locale` mandatory, `--redacted` keeps the verdicts and drops the payload. With no baseline yet it renders the run on its own and says so, so the first run is readable before anything is promoted |
| `digline explain` | the same facts read back at length, in prose: what ran, what moved, against which measured interval, what differed underneath. Compares when there is a baseline and reads the run alone when there is not. `--json` emits the fact list the prose is rendered from. It states, and never advises — [`docs/explain.md`](docs/explain.md) |
| `digline log` | which model answered, read down the stored runs: what each side sent, what the provider said answered, the rolls between them, and every run that recorded nothing named for what it is. A roll is declared by the record, never deduced from scores. Never a gate: it exits 0 whatever it finds — [`docs/log.md`](docs/log.md) |
| `digline rejudge` | judge a stored run's recorded answers again — a changed judge, rubric or threshold, over the same answers, at no cost to the target. The run it writes declares where the answers came from and cannot be promoted — [`docs/rejudge.md`](docs/rejudge.md) |
| `digline list` | stored runs, newest first, baseline marked |
| `digline view` | local browser UI — [`docs/view.md`](docs/view.md) |
| `digline migrate` | bring stored runs forward across schema versions — [`docs/migrate.md`](docs/migrate.md) |

The same comparison reaches an agent through [`digline-mcp`](docs/mcp.md) —
eight tools that read and measure, and no `promote` to call — a test run through
[`pytest-digline`](docs/pytest.md), one row per check, a pull request through
[`digline/digline-action`](https://github.com/digline/digline-action), and CI
without a Python toolchain through `ghcr.io/digline/digline`.

## Examples

Ten projects in [`examples/`](examples/), each answering a question somebody
actually arrives with. Every one runs with no API key, carries its committed
`report.html`, and is a standalone project: copy the directory anywhere and
`uv sync` works.

- [**I have a classifier: how do I keep it under control?**](examples/classifier/) — labelled cases, an agreement check, `Precision` and `Accuracy` as the gate
- [**I'm writing a prompt and have no application yet**](examples/prompt-first/) — a prompt in a file, and the report showing its diff next to what it moved
- [**I have a RAG: how do I check it doesn't make things up?**](examples/rag/) — frozen retrieval, `Faithfulness`, `PiiAbsent`
- [**My application is Java: can I use this?**](examples/external-app/) — `HttpTarget` against a service digline cannot import
- [**My app is LangChain4j: what do I put in my repo?**](examples/langchain4j/) — the walkthrough: one endpoint, three files, the CI gate
- [**My pipeline is LangChain: what changed when I upgraded it?**](examples/langchain/) — the chain called in process, `FakeListChatModel` in CI, one line to a real model
- [**My agent calls the right tools, but with the right arguments?**](examples/langgraph/) — a LangGraph agent judged on its trajectory: `ToolsCalled` for the order, `ToolCalledWith` for the arguments, the tools real and the model scripted
- [**My RAG is LlamaIndex: is it still answering from the right page?**](examples/llamaindex/) — a live query engine, retrieval measured by `Faithfulness` against the page each case declares
- [**My team does not write Python: can we still gate a prompt?**](examples/quickstart-toml/) — a `suite.toml` and a `cases.json`, no code in the suite
- [**My suite is green today: who watches it on Thursday?**](examples/operator/) — the operator loop: a scheduled re-run, draw told from drift, a declared policy that decides who is woken, and a probe proving each cycle that `promote` is still absent

## What digline is not

- **Not an observability platform.** Dashboards over production traces are a
  served market. What is designed and not yet built is narrower: evaluating
  production responses inside *your* perimeter, and turning a failure into a
  committed test case.
- **Not a red-teaming tool.** digline generates no attacks. Once one is found,
  it becomes a `Case`, and the suite makes sure it never works again.
- **Not YAML.** The suite is Python — or, within declared limits, TOML: cases
  were always data, and now the rules can be too. A judge with rules of its own,
  a computed request body and a custom assertion stay Python, and the loader
  names the wall you hit rather than half-supporting it.
- **Not a funnel.** Two commitments, by design and for good: no hosted service
  that receives your payloads, and no data collection. The baseline lives in
  your repo; the runs happen on your machines. If digline ever grows paid
  features, they will run inside your perimeter too. How digline treats its own
  attack surface — the published advisories and the delta-pass every minor
  release gets — is in [`SECURITY.md`](SECURITY.md).

## Status

`0.15.0`, pre-1.0. The offline cycle — write the suite, run, compare, promote,
report, and commit what a person decided — is complete, covered by tests, and
used daily on a real project, and
since 0.5.0 the suite may be written as data as well as in Python. The API may
still change before 1.0; the baseline format is versioned and migrates. The
production store, the bridge from production failures back to committed cases,
and the reactive side are designed in
[ADR 0002](docs/adr/0002-three-worlds-and-where-the-data-lives.md) and not
written yet.

Python 3.12+. One runtime dependency: `jsonschema`.

## Docs

- [`docs/guide.md`](docs/guide.md) — how to reason with digline, in eight chapters
  and the order the problems arrive: baseline, judge noise, sampling, tolerance,
  threshold, which run to promote, what to gate on, what to maintain
- [`docs/metrics.md`](docs/metrics.md) — a card per assertion and aggregate: when
  to reach for it, what it produces, what it will do to you if you are not looking
- [`docs/api.md`](docs/api.md) — what is imported from where, every assertion
  and its parameters, custom assertions, and the complete example in
  [`examples/quickstart/`](examples/quickstart/), which a test runs on every build
- [`docs/declarative.md`](docs/declarative.md) — the suite as data: the TOML
  format, key by key, what it deliberately cannot say, and how to move a suite
  between the two forms without losing its baseline
- [`docs/diff.md`](docs/diff.md) · [`docs/explain.md`](docs/explain.md) · [`docs/rejudge.md`](docs/rejudge.md) · [`docs/view.md`](docs/view.md) · [`docs/migrate.md`](docs/migrate.md) — each command with a surface of its own
- [`docs/mcp.md`](docs/mcp.md) · [`docs/pytest.md`](docs/pytest.md) — the two front ends that are not the CLI
- [`AGENTS.md`](AGENTS.md) — how a coding agent should operate digline in your repo
- [`docs/adr/`](docs/adr/) — the architectural decisions, numbered, with the reasoning

## License

Apache-2.0.
