Metadata-Version: 2.5
Name: nullcov
Version: 0.1.0
Summary: Honest coverage for agent scaffolds: unexercised layers report unknown, never 100%.
License: MIT License
        
        Copyright (c) 2026 Shay
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agents,coverage,evaluation,llm,pytest,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: pytest>=8.0
Provides-Extra: dev
Requires-Dist: langgraph>=0.2; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: langgraph
Requires-Dist: langgraph>=0.2; extra == 'langgraph'
Description-Content-Type: text/markdown

# nullcov

**Your agent test suite reports 94%. This tells you which part of that number is a lie.**

Conventional coverage divides passing cases by total cases. A layer of your agent
that nobody wrote a case for contributes nothing to either side of that division —
so it vanishes from the summary entirely. The suite looks strongest exactly where
it is blindest.

nullcov reports those layers as `unknown`. Never `100%`, never silently dropped.

```
============================= nullcov ==============================
  [ok  ] routing                  3 slice(s), 41 case(s)
  [ok  ] schema                   2 slice(s), 18 case(s)
  [????] escalation               no cases -- coverage unknown, not 100%
  [FAIL] safety                   1 slice(s), 6 case(s), 1 failing
  [----] reasoning                generative -- not scored, by design

  65 case(s) ran, 1 failing (98.5% of executed cases passed).
  1 declared layer(s) have no coverage. The number above says nothing about them.
```

## Why this exists

In a June 2026 survey of 157 enterprises (VentureBeat VB Pulse), **half had shipped
an agent that passed their own evaluations and still failed in front of a customer.**
Only 5% fully trusted the evaluations they were shipping on. When asked what was
wrong with those evaluations, the most common answer was not "not enough coverage" —
it was **poor alignment with real-world outcomes**.

The same pattern shows up in the research. A production agent team
([arXiv:2606.11686](https://arxiv.org/abs/2606.11686)) documented a confirmation-gate
bug that caused roughly half of confirmed orders to silently not execute — while
their aggregate quality metric barely moved. Under controlled fault injection,
end-to-end pass rates dropped 1.7–5.9% while layer-isolated slices dropped 25–91%.

One aggregate number hides a broken layer. That is the problem nullcov is built for.

## Install

```bash
pip install nullcov
```

## Use

**1. Declare your layers.** In `pyproject.toml` (or a `nullcov.toml`):

```toml
[tool.nullcov]
layers = ["routing", "schema", "escalation", "safety"]
```

This declaration is the point. Without a statement of what *should* exist, an
untested layer is indistinguishable from a layer that does not exist, and no tool
can tell you the difference. Keeping it in version control means deleting a layer
to make a report go green is a diff a reviewer sees — not a number that quietly
improves.

Layers that are genuinely generative are declared as such and are never scored:

```toml
[tool.nullcov.layers]
routing = { description = "picks the next node" }
reasoning = { deterministic = false }
```

**2. Attach tests to layers.**

```python
import nullcov

@nullcov.case(layer="escalation", slice_id="high_value_requires_human")
def test_high_value_transactions_go_to_a_human():
    with nullcov.pure_mode():
        result = run_scaffold(graph, {"amount": 1200})
    assert result.next_node == "human_review"
```

**3. Run.**

```bash
pytest --nullcov
```

Add `--nullcov-strict` to make it a CI gate. An unexercised declared layer fails
the run, because a suite that cannot speak to a layer has not verified it.

## Pure mode

`pure_mode()` blocks outbound network connections at the socket layer for the
duration of a block. It is not a stub you have to remember to install — it holds
regardless of which SDK the agent reaches for, and raises `PureModeViolation`
naming the host if anything tries to escape.

```python
with nullcov.pure_mode() as report:
    result = graph.invoke(state)
assert report.clean
```

A test that passes under pure mode **provably** made zero model calls. Not by
convention, not because someone remembered to patch the client — because nothing
could reach the network. That makes it deterministic, free, and fast enough to run
on every commit. Loopback stays open, so local fixtures and recorded-cassette
servers still work.

## Reality alignment

Coverage tells you what you looked at. It cannot tell you whether you looked at the
right thing. Feed nullcov your production incidents and it separates two failures
that look identical on a dashboard:

```bash
pytest --nullcov --nullcov-incidents incidents.json
```

```
reality alignment
  29 attributed incident(s).

  Blind spots -- production fails here and no case looks:
    escalation: 11 incidents (38% of all attributed) and no coverage at all.
    The suite cannot speak to this layer.

  Misaligned -- cases pass here and production fails anyway:
    routing: 9 incidents (31% of all attributed) despite 41 passing cases.
    The cases pass and production still fails -- distrust these cases before
    adding more.
```

**Blind spot**: no coverage. Write cases.

**Misalignment**: full, passing coverage — and production fails there anyway. More
cases of the same shape make this worse, not better. The existing cases are the
thing to distrust. This is the finding that explains a green pipeline and a
customer-visible outage on the same afternoon, and it is the reason this tool is
not just another eval runner.

`incidents.json` is a list of records — an export from your incident tracker, or a
handful of postmortem entries:

```json
[
  {"incident_id": "INC-4471", "layer": "escalation", "severity": "sev2",
   "summary": "refund over threshold auto-approved"}
]
```

Attribution is the one input nullcov cannot derive for itself, and the quality of
everything above depends on it.

## Structural coverage for LangGraph

Everything above depends on a human typing a truthful layer name. Nothing stops
that from drifting: add a node to the graph, forget to tag a test for it, and
it is invisible everywhere -- not `[????]`, not undeclared, just absent.

`nullcov.langgraph` removes the human from that specific step. It reads the
topology straight off your **compiled** graph object -- the actual nodes and
conditional branches LangGraph built, not a file you maintain by hand -- and
tells you which of them any test run ever actually traversed.

```python
from nullcov.langgraph import CoverageAccumulator, GraphTopology

graph = build_graph()  # your compiled LangGraph graph
accumulator = CoverageAccumulator(topology=GraphTopology.extract(graph))

# in your tests, run the graph through the accumulator instead of graph.invoke()
accumulator.record_run(graph, {"amount": 50, "reason": "wrong size"})
accumulator.record_run(graph, {"amount": 600, "reason": "wrong size"})

coverage = accumulator.coverage()
print(coverage.branch_rate())        # 0.5
print(coverage.untaken_branches)     # {Branch("validate", "reject")}
print(coverage.unvisited_nodes)      # {"reject"}
```

Run that against the example agent in this repo and it finds a real gap the
manual layer system misses entirely: `validate()` has a passing unit test for
an invalid payload, and `reject` is a fully-implemented node — but no test ever
runs the **compiled graph** end-to-end with a bad input, so the
`validate → reject` transition has never once fired in the test suite. A unit
test on the routing function proves the function is correct. It proves nothing
about whether the graph, as wired, ever takes that path.

We checked before building this: neither AWS Bedrock AgentCore Evaluations nor
`fasteval-langgraph` (Intuit) compute this. Both score individual test cases
against a trajectory the developer writes by hand — a real capability, but a
different one. Neither aggregates, across a whole suite, what fraction of the
graph's actual branch topology was ever exercised. `nullcov.langgraph` is
narrower than either of them and answers a question they don't ask.

Requires `langgraph` (`pip install nullcov[langgraph]`). This module is
LangGraph-specific by design — the ground truth it reads only exists because
`CompiledGraph.get_graph()` exposes it. CrewAI and AutoGen would each need
their own adapter reading their own internal representation; none exists yet.

## What this does not do

- **It does not score generative output.** LLM-as-judge is not statistically
  calibrated — position, verbosity and self-enhancement bias are documented, and
  calibrating judge confidence needs logprobs most providers do not expose. nullcov
  declines to put a number on free-form reasoning rather than pretending.
- **It does not prove your agent is correct.** It proves which contracts held and,
  more usefully, which ones nobody checked.
- **It does not replace your eval suite.** It tells you where that suite is blind.

## Prior art

nullcov leans on published work rather than inventing around it. The
scaffold/generative split and the coverage-honesty criterion come from
[arXiv:2606.11686](https://arxiv.org/abs/2606.11686). Deterministic record-and-replay
of agent I/O is solved by [agrepl](https://arxiv.org/abs/2607.16200) and
`vcr-langchain`; multi-agent tracing by
[MAESTRO](https://github.com/sands-lab/maestro). Signed action receipts exist in
`nobulex` and `protect-mcp`. If you need those, use those.

What nullcov adds is the honesty constraint on the aggregate, and the alignment
pass against production.

## License

MIT
