Metadata-Version: 2.4
Name: invarlock
Version: 0.16.1
Summary: Paired model release-regression evaluation with independently verifiable evidence
Author-email: InvarLock Team <oss@invarlock.dev>
Maintainer-email: InvarLock Maintainers <support@invarlock.dev>
License-Expression: Apache-2.0
Project-URL: Homepage, https://invarlock.ai
Project-URL: Repository, https://github.com/invarlock/invarlock
Project-URL: Documentation, https://invarlock.github.io/invarlock/0.16.1/
Project-URL: Issues, https://github.com/invarlock/invarlock/issues
Project-URL: Changelog, https://github.com/invarlock/invarlock/blob/v0.16.1/CHANGELOG.md
Keywords: machine-learning,evaluation,verification,evidence,model-assurance,release-regression,reproducibility
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.24.1
Requires-Dist: click>=8.3.3
Requires-Dist: cryptography>=50.0.0
Requires-Dist: rich>=13.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: jsonschema>=4.0
Provides-Extra: judge
Requires-Dist: invarlock-inspect-judge[inspect]==0.16.1; extra == "judge"
Provides-Extra: dev
Requires-Dist: pytest>=9.1.1; extra == "dev"
Requires-Dist: pytest-cov>=7.1.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.6; extra == "dev"
Requires-Dist: pillow<13,>=11.3; extra == "dev"
Requires-Dist: ruff==0.16.7; extra == "dev"
Requires-Dist: mypy==2.3.1; extra == "dev"
Requires-Dist: types-jsonschema>=4.26; extra == "dev"
Requires-Dist: hypothesis>=6.98; extra == "dev"
Requires-Dist: pre-commit>=4.6.1; extra == "dev"
Requires-Dist: markdown>=3.10.2; extra == "dev"
Requires-Dist: markdown-it-py>=4.0.0; extra == "dev"
Requires-Dist: mkdocs>=1.5; extra == "dev"
Requires-Dist: mkdocs-material>=9.7.7; extra == "dev"
Requires-Dist: mkdocs-mermaid2-plugin>=1.2.3; extra == "dev"
Requires-Dist: pymdown-extensions>=11.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: wheel>=0.48.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: readme-renderer[md]==45.0; extra == "dev"
Provides-Extra: ci
Requires-Dist: setuptools>=83.0.0; extra == "ci"
Requires-Dist: pytest>=9.1.1; extra == "ci"
Requires-Dist: pytest-cov>=7.1.0; extra == "ci"
Requires-Dist: pytest-xdist>=3.6; extra == "ci"
Requires-Dist: pillow<13,>=11.3; extra == "ci"
Requires-Dist: hypothesis>=6.98; extra == "ci"
Requires-Dist: ruff==0.16.7; extra == "ci"
Requires-Dist: mypy==2.3.1; extra == "ci"
Requires-Dist: types-jsonschema>=4.26; extra == "ci"
Requires-Dist: build>=0.10.0; extra == "ci"
Requires-Dist: markdown>=3.10.2; extra == "ci"
Requires-Dist: markdown-it-py>=4.0.0; extra == "ci"
Requires-Dist: wheel>=0.48.0; extra == "ci"
Requires-Dist: twine>=4.0.0; extra == "ci"
Requires-Dist: readme-renderer[md]==45.0; extra == "ci"
Provides-Extra: docs-ci
Requires-Dist: linkchecker>=10.6.0; extra == "docs-ci"
Requires-Dist: mkdocs>=1.5; extra == "docs-ci"
Requires-Dist: mkdocs-material>=9.7.7; extra == "docs-ci"
Requires-Dist: mkdocs-mermaid2-plugin>=1.2.3; extra == "docs-ci"
Requires-Dist: pymdown-extensions>=11.0.0; extra == "docs-ci"
Provides-Extra: precommit-ci
Requires-Dist: pre-commit>=4.6.1; extra == "precommit-ci"
Provides-Extra: release-ci
Requires-Dist: build>=0.10.0; extra == "release-ci"
Requires-Dist: setuptools>=83.0.0; extra == "release-ci"
Requires-Dist: wheel==0.48.0; extra == "release-ci"
Requires-Dist: twine>=4.0.0; extra == "release-ci"
Requires-Dist: readme-renderer[md]==45.0; extra == "release-ci"
Requires-Dist: cyclonedx-bom>=4.1; extra == "release-ci"
Provides-Extra: security-ci
Requires-Dist: pip>=26.2; extra == "security-ci"
Requires-Dist: pip-audit>=2.8; extra == "security-ci"
Requires-Dist: msgpack>=1.2.1; extra == "security-ci"
Requires-Dist: cyclonedx-bom>=4.1; extra == "security-ci"
Dynamic: license-file

<p align="center">
  <img
      src="https://raw.githubusercontent.com/invarlock/invarlock/v0.16.1/docs/assets/invarlock-logo.svg"
      alt="InvarLock"
      width="420"
    />
</p>

<p align="center"><em>Evaluate model changes. Verify the evidence. Share the result.</em></p>

<p align="center">
  <a href="https://github.com/invarlock/invarlock/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/invarlock/invarlock/ci.yml?branch=main&label=CI&logo=github&labelColor=18150f" /></a>
  <a href="https://pypi.org/project/invarlock/"><img alt="PyPI" src="https://img.shields.io/pypi/v/invarlock?label=PyPI&logo=pypi&labelColor=18150f&color=1f3a7a" /></a>
  <a href="https://invarlock.github.io/invarlock/"><img alt="Docs" src="https://img.shields.io/badge/docs-quickstart-1f3a7a?labelColor=18150f" /></a>
  <a href="https://github.com/invarlock/invarlock/blob/v0.16.1/LICENSE"><img alt="License: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-1f3a7a?labelColor=18150f" /></a>
  <a href="https://www.python.org/downloads/release/python-3120/"><img alt="Python 3.12+" src="https://img.shields.io/badge/python-3.12%2B-1f3a7a?logo=python&logoColor=f4efe3&labelColor=18150f" /></a>
</p>

**InvarLock evaluates model changes and produces evidence that another team can
verify independently.** Compare a candidate (the *subject*) with an approved
baseline, using tests and acceptance thresholds you choose. Run a supported
comparison through InvarLock, or bring per-case records from your existing
evaluation workflow.

The result is a signed evidence bundle, a separate verification receipt and a
report explaining the comparison. Your customer or internal reviewer can check
the retained result against agreed inputs and policy without rerunning model
inference.

- **Three native scorers:** exact match, normalized NLL and bounded LLM judging.
- **Existing workflows:** supported evaluator exports and a Python capture SDK.
- **Offline verification:** replay the supported analysis with recipient-owned
  trust inputs; no provider credentials needed.
- **Reviewable results:** model and service identities, measured changes,
  uncertainty bounds and the checks that passed or failed.

## Quickstart

Try a signed evidence check on a regular CPU. **Python 3.12+** is required;
no GPU, model download, API key or container engine is needed for this example.

From a checkout of this repository:

```bash
python -m venv .venv
. .venv/bin/activate
python -m pip install .
python examples/quickstart/run.py \
  --fixture examples/acceptance-handoff/golden
```

The command verifies retained evidence against the example's separate trust
inputs, issues a new signed receipt and writes an HTML report. It prints:

```text
PASS signed evidence verified
Decision: pass
```

Open `invarlock-quickstart-output/evidence.html`. The same directory contains
`verification.receipt.json` and `verification.result.json`. This demonstrates
verification and reporting of a fixed comparison; it makes no new model calls.

<details>
<summary>Use a published wheel instead</summary>

Start in an empty directory and download the examples matching the installed
release:

```bash
python -m venv .venv
. .venv/bin/activate
python -m pip install invarlock
INVARLOCK_VERSION="$(python -c 'from importlib.metadata import version; print(version("invarlock"))')"

curl -fsSLO \
  "https://github.com/invarlock/invarlock/archive/refs/tags/v${INVARLOCK_VERSION}.tar.gz" &&
tar -xzf "v${INVARLOCK_VERSION}.tar.gz" --strip-components=3 \
  "invarlock-${INVARLOCK_VERSION}/examples/quickstart" \
  "invarlock-${INVARLOCK_VERSION}/examples/acceptance-handoff/golden" &&

python run.py --fixture golden
```

</details>

Use documentation and examples from the same release as a published wheel.
For local builds, use the exact source checkout that built the package; retain
its commit and wheel digest when sharing it. A missing release archive is an
error, never a reason to substitute another version. See
[matching wheels and examples](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/getting-started.md#matching-wheels-and-examples).

To collect new judge ratings, install `python -m pip install "invarlock[judge]"`.
Judge scoring is built into the core; this extra adds the matching collector and
its pinned provider SDKs. Offline import, verification and reporting need only
`invarlock`. See [judge setup](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/judge-measurements.md#frozen-answer-requests-and-preflight).

## What can you use it for?

| Your task | Starting point |
| --- | --- |
| Check a fine-tune, quantized model or runtime change against a baseline | [Model-change workflows](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/change-scenarios.md) |
| Add a verifiable comparison to an existing evaluator or CI pipeline | [Captured results](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/captured-results.md) |
| Grade frozen answers under a task-specific rubric | [Judge scoring](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/judge-measurements.md) |
| Recheck a hosted service after a change or on a schedule | [Hosted-service requalification](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/hosted-service-requalification.md) |
| Send evidence to a customer or internal release reviewer | [Evidence and verification](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/evidence-and-verification.md) |

These workflows fit teams that repeatedly need to produce, check or retain
evidence supporting a model change: model suppliers, fine-tuning and optimization
teams, and internal AI teams with a release-review process.

## One workflow: evaluate, verify, report

<p align="center">
  <img
    src="https://raw.githubusercontent.com/invarlock/invarlock/v0.16.1/docs/assets/evaluation-verification-flow.svg"
    alt="Native comparisons, captured records and frozen answers feed evaluation; independent verification and reporting use the resulting evidence"
    width="100%"
  />
</p>

For a captured deterministic comparison, prepare a request, signing key and
independent trust profile:

```bash
invarlock evaluate request.yaml --signing-key signing-key.pem --preflight --json
invarlock evaluate request.yaml --signing-key signing-key.pem
invarlock verify evidence/ --trust-profile trust/trust-inputs.json \
  --receipt verification.receipt.json
invarlock report evidence/ --html report.html
```

Use the evidence destination declared in your request. The
[captured-results guide](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/captured-results.md)
provides complete setup instructions, keys and trust-profile preparation.
Native runs also need runtime resources; judge receipt issuance uses its own
verifier-key and identity options. Follow the linked guide for that workflow.

| Command | What it does |
| --- | --- |
| `evaluate` | Runs or imports a declared comparison, applies policy and publishes evidence. Preflight checks setup without execution. |
| `verify` | Checks signatures, bound identities and supported analysis against the recipient's independent expectations; can issue a signed receipt. |
| `report` | Explains the identities, results, uncertainty and policy checks in the retained evidence. Rendering does not replace independent verification. |

The HTML report supports multiple metrics and slices, with an overview and
per-result detail. Terminal and Markdown outputs support review; JSON and JUnit
support automation where the selected workflow provides them. See the
[CLI reference](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/cli.md)
for output options and policy exit codes.

## Choose a scorer

The same native and captured entry points support three built-in scorers, each
with its own required observations and statistical treatment:

| Scorer | Required observations | Comparison |
| --- | --- | --- |
| Exact match (`exact_match`) | Answers and reference answers for paired cases | Accuracy change with a paired uncertainty interval |
| Normalized NLL (`normalized_nll_per_utf8_byte`) | Reference-continuation log probabilities, UTF-8 byte counts and bound tokenizer metadata | Ratio of mean byte-normalized NLL, with paired resampling |
| Judge (`judge`) | Frozen task text and answers, declared rubric and retained judge calls | Bounded ratings aggregated by declared independent units |

Policy uses the conservative uncertainty bound, with the configured sample,
precision and quality requirements. A point estimate alone does not decide the
result. [Schedule and policy](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/schedule-and-policy.md)
explains the statistical scope and thresholds.

Judge collection uses the optional `invarlock-inspect-judge[inspect]` package
with explicit call, token, cost, timeout and checkpoint limits. Importing retained
ratings, verification and reporting work offline in the core wheel. Start with
`invarlock evaluate --init my-judge --example native-judge` and replace the
illustrative model pins with your actual runtime inputs. See
[judge scoring](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/judge-measurements.md)
for the supported rubric, reference and collection profiles.

Task-specific deterministic extensions cover normalized labels, numeric
tolerances, structured fields and token overlap.
[Evidence sets](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/evidence-sets.md)
can require multiple complementary checks on the same frozen answers while
preserving each check's meaning.

## Keep your evaluator or run the comparison here

**Use existing records.** Installed parsers cover selected Inspect AI, LM
Evaluation Harness and Promptfoo export profiles, plus canonical JSON and generic
JSONL. The [capture SDK](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/api-guide.md)
lets other pipelines supply the required per-case facts. Structured inputs need
an explicit text projection for judging; normalized NLL needs actual likelihood
measurements. An aggregate score cannot substitute for missing observations.
The [qualification matrix](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/evaluator-qualification.md)
distinguishes installed support, replay authority and retained runtime evidence
for each declared profile.

**Run a native comparison.** Hugging Face Transformers is the built-in runtime.
Optional GGUF/llama.cpp, TensorRT-LLM and Hugging Face vision-text packages provide
additional runtime profiles. Native run mode uses a caller-authorized,
digest-addressed Docker or Podman image. Follow the
[getting-started guide](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/getting-started.md)
for artifact pins, runtime setup and independent verification inputs.
Host-side model preparation uses the matching checkout's
[HF runtime group](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/runtime-providers.md#hugging-face-transformers)
after bootstrapping its verified hardened Accelerate wheel.
The [import request](https://github.com/invarlock/invarlock/blob/v0.16.1/examples/request.yaml)
uses complete retained provider sidecars and omits `--runtime-image` and `--runtime-image-digest`.

**Recheck a hosted service.** Your harness records fresh executions, service
configuration and observation windows. InvarLock compares those captured facts
and verifies the resulting evidence offline. Hosted identity describes the
observed service; it does not claim access to hidden model weights. Your scheduler
initiates periodic comparisons. Verifying old evidence does not measure the
service again.

## Inspect real retained examples

The repository includes signed evidence, receipts and replay instructions from
actual model runs. Each reference establishes its declared workflow and scope:

| Reference | Retained work |
| --- | --- |
| [Native model and runtime comparisons](https://github.com/invarlock/invarlock/tree/v0.16.1/public_evidence) | Pinned text, GGUF, vision-text and TensorRT-LLM comparisons |
| [Evaluator handoffs](https://github.com/invarlock/invarlock/blob/v0.16.1/examples/evaluator-qualification/signed-transactions/README.md) | 400-record Qwen3.5 9B Harness and Inspect journeys; a Gemma instruction-to-QAT comparison |
| [Hosted HTTP capture](https://github.com/invarlock/invarlock/blob/v0.16.1/examples/hosted-service/references/mistral-7b-http/README.md) | 400 paired cases from distinct Mistral 7B base and instruction checkpoints behind a local HTTP service |
| [Likelihood comparison](https://github.com/invarlock/invarlock/blob/v0.16.1/examples/captured-results/references/mistral-7b-likelihood/README.md) | Distinct Mistral 7B checkpoints on 400 fixed narrative continuations |
| [Bounded judge comparison](https://github.com/invarlock/invarlock/blob/v0.16.1/examples/judge-measurements/references/k2-32b-luna-xhigh-heldout/README.md) | 10,260 retained ratings across 1,710 QA and extraction cases, with offline replay |

A comparative pass is not proof of adequate task quality or representative
production performance. The local HTTP example does not qualify an external
provider; repeated judge ratings are not additional independent cases.

## What verification establishes

A recipient supplies its expected identities, policy and signer trust through a
channel independent of the submitted bundle. InvarLock checks the package and
reconstructs the supported analysis against those expectations. Native runtime
bindings and imported observations retain different provenance claims.

Verification checks retained evidence; it does not independently rerun the
original model execution. It does not establish that the chosen benchmark covers
production traffic, that a rubric captures every requirement, or that a passing
comparison authorizes deployment. Read the
[assurance case](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/assurance/assurance-case.md)
and [trust model](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/security/trust-model.md)
for the precise guarantees and assumptions.

For artifact-delivery automation, optional
[acceptance attestations](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/acceptance-attestations.md)
and [OPA/Rego or CUE policies](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/policy-engine-interop.md)
consume the authenticated result under recipient-controlled policy.
v0.13 evidence and receipts remain verifiable and ingestible; acceptance always
uses the recipient's current policy. InvarLock is pre-1.0, with explicit artifact
format versions and a Python API that may evolve between minor releases.

## Documentation and contributing

- [Getting started](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/user-guide/getting-started.md) · [Examples](https://github.com/invarlock/invarlock/tree/v0.16.1/examples) · [CLI](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/cli.md) · [Python API](https://github.com/invarlock/invarlock/blob/v0.16.1/docs/reference/api-guide.md)
- [Contributing](https://github.com/invarlock/invarlock/blob/v0.16.1/CONTRIBUTING.md) for development setup and required checks.
- [Discussions](https://github.com/invarlock/invarlock/discussions) for questions and integration ideas; [Issues](https://github.com/invarlock/invarlock/issues) for reproducible bugs.
- [Security policy](https://github.com/invarlock/invarlock/blob/v0.16.1/SECURITY.md) for private vulnerability reports.

Apache-2.0. See [LICENSE](https://github.com/invarlock/invarlock/blob/v0.16.1/LICENSE)
and [third-party notices](https://github.com/invarlock/invarlock/blob/v0.16.1/THIRD_PARTY_NOTICES.md)
for dependency and retained-data terms.
