Metadata-Version: 2.5
Name: acvp-assay
Version: 0.23.1
Summary: Run NIST ACVP vectors against any implementation and catch conformance regressions
Project-URL: Homepage, https://github.com/Govardhan527/acvp-assay
Project-URL: Repository, https://github.com/Govardhan527/acvp-assay
Project-URL: Changelog, https://github.com/Govardhan527/acvp-assay/blob/main/CHANGELOG.md
Project-URL: Harness protocol, https://github.com/Govardhan527/acvp-assay/blob/main/docs/harness-protocol.md
Project-URL: Commercial support, https://github.com/Govardhan527/acvp-assay/blob/main/SERVICES.md
Project-URL: Issues, https://github.com/Govardhan527/acvp-assay/issues
Author-email: Govardhan Yadava <govardhan@seccrypto.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: acvp,cavp,conformance,cryptography,fips-140-3,test-vectors
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security :: Cryptography
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: cryptography<51,>=50.0.1
Provides-Extra: dev
Requires-Dist: build<2,>=1.6.0; extra == 'dev'
Requires-Dist: mypy<3,>=2.3.1; extra == 'dev'
Requires-Dist: pytest-cov<8,>=7.1.0; extra == 'dev'
Requires-Dist: pytest<10,>=9.1.1; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.16.5; extra == 'dev'
Description-Content-Type: text/markdown

# ACVP Assay

[![CI](https://github.com/Govardhan527/acvp-assay/actions/workflows/ci.yml/badge.svg)](https://github.com/Govardhan527/acvp-assay/actions/workflows/ci.yml)
[![Verified against NIST ACVTS](https://img.shields.io/badge/NIST%20ACVTS%20Demo-59%2F59%20algorithms%20passed-2ea44f)](#verified-against-nists-own-server)
[![Coverage](https://img.shields.io/badge/coverage-99.7%25-2ea44f)](#development-commands)

> ## ✅ Judged by NIST's own server
>
> **All 59 supported algorithm names** have been run against **vectors NIST generated live** and
> submitted back for NIST to judge. On `demo.acvts.nist.gov`, that is **65 vector sets and 57,116
> test cases across 21 completed sessions**, where the two largest sessions are 53.4% of the cases
> (partitioned cases-by-session; a flat split across 21 would be 9.5%). **Every one of those sets
> came back `"passed"`**, and that verdict is the server's, not this project's.
> [Exactly which, per algorithm](#coverage).
>
> Read those two numbers precisely: ACVP returns one verdict **per vector set**, so 65 is the
> count of verdicts NIST issued and 57,116 is the number of cases inside them. The server never
> issued 57,116 separate verdicts, and this project does not claim it did.
>
> **What this does and does not say about ML-KEM and ML-DSA.** Those two sessions were answered
> by `examples/pqc_reference_harness.py`, which is backed by `kyber-py` and `dilithium-py` —
> educational implementations that are **not constant-time and make no side-channel claims**.
> The passing verdict is evidence that *this runner* parses, routes and answers PQC vector sets
> correctly. It is emphatically **not** evidence that any implementation is fit to ship, and
> there is no built-in PQC provider here: for post-quantum work the implementation under test is
> yours, supplied through `--provider-command`.
>
> Most tools of this kind are checked against static files only. Being checked by the system
> that issues the vectors is what caught the six defects listed under
> [What this caught that fixtures did not](#what-this-caught-that-fixtures-did-not).
>
> **This is test evidence, not a certificate.** It confers no validation status: only an
> accredited CST or 17ACVT laboratory performs CAVP or FIPS 140-3 validation, and Demo is not
> the production ACVTS.

Run NIST ACVP test vectors against any implementation — including ones you cannot link against — and catch conformance regressions between runs.

An *assay* measures composition, it does not certify it. This tool produces reproducible test evidence: it parses ACVP vector sets, executes them through a replaceable provider boundary, compares results case by case, and diffs one run against the next. It does **not** perform or substitute for CAVP algorithm validation or FIPS 140-3 cryptographic-module validation — only accredited CST and 17ACVT laboratories can do that.

Two things distinguish it from `libacvp` and ACVP Proxy, which cover more algorithms and speak the live protocol:

- **It reaches implementations they cannot.** A harness reads one JSON request on stdin and writes one on stdout, so an HSM, a smartcard, an embedded device over a serial link, or a library in any language can be tested without linking anything.
- **It answers "are we *still* conformant?"** `acvp-assay diff` compares two runs and reports regressions, including coverage that silently disappeared. With re-validation running well over a year, that is the failure mode that costs a cycle.

## Commercial support

The tool is MIT-licensed and free, and nothing here is gated. If you would rather have the work
done than the tool handed over — a readiness assessment before you engage a laboratory, a harness
built against your HSM or embedded target, or conformance regression wired into your CI — see
[SERVICES.md](SERVICES.md).

Written and maintained by Govardhan Yadava: seven years owning cryptographic validation for an
enterprise HSM and key-management platform, FIPS 140-2/140-3 mode behaviour across a five-client
matrix, and twelve review comments submitted to the OASIS KMIP Profiles v3.0 public review
(August 2026). Contact: govardhan@seccrypto.dev

## Scope

Implemented today:

- **59 algorithm names across 34 families** — AES in GCM, CCM, ECB, CBC, CTR, OFB, CFB128,
  CFB8, CFB1, CBC-CS1/CS2/CS3,
  GMAC, KW, KWP and XTS; CMAC-AES; all three SP 800-90A DRBGs; KDF SP 800-108; KDA SP 800-56C;
  SHA-1, SHA-2, SHA-3 and the SHAKE XOFs; PBKDF; safePrimes;
  the SSH KDF, TLS 1.2 and TLS 1.3; KAS-IFC-SSC and KTS-IFC;
  HMAC over each; RSA; ECDSA; KAS-ECC-SSC and KAS-FFC-SSC; ML-KEM and ML-DSA
- a replaceable provider boundary, in-process or an external harness over JSON — **all 59
  names reach a harness**, so nothing silently tests this project's OpenSSL binding when you
  asked for your own implementation
- **live ACVTS submission from your implementation**: `acvts_client.py submit
  --provider-command ...` answers NIST-generated vectors from your code and returns NIST's
  verdict — all 59 names, ML-KEM and ML-DSA included
- run-over-run regression diffing, including coverage that silently disappeared
- typed parsing that preserves `vsId`, `tgId`, and `tcId`
- deterministic tests on Linux, verified against pinned NIST vectors, and for **all 59 names
  against vectors generated live by NIST's ACVTS server** — see [Coverage](#coverage)

Deliberately out of scope: a general-purpose ACVP protocol client (`libacvp` and
ACVP Proxy already do that well), an HTML dashboard, performance benchmarking,
and redistribution of upstream vectors whose licensing is unconfirmed. There is
also no built-in ML-KEM or ML-DSA implementation — for post-quantum work the
implementation under test is yours, supplied through `--provider-command`.

The one piece of protocol code in the repository, `scripts/acvts_client.py`, is
not an exception to that: it exists so the runner can be checked against NIST's
live server rather than against a fixture, and it implements only what that
needs — register, fetch, submit, results. See
[Verified against NIST's own server](#verified-against-nists-own-server).

Versions before 1.0.0 do not promise a stable provider API; the protocols are
still settling as algorithm families are added.

## Requirements

- Python 3.12 or newer
- Internet access during setup to download pinned dependency ranges
- A Python installation with `venv` support

On Debian or Ubuntu, `venv` may be a separate package such as `python3.12-venv`.

## Quick start

```bash
python3.12 scripts/dev.py setup
python3.12 scripts/dev.py test
python3.12 scripts/dev.py demo
.venv/bin/python -m acvp_assay run fixtures/aes-gcm-valid-encrypt/prompt.json
```

The demo prints machine-readable runtime metadata: the `cryptography` and OpenSSL versions that identify the provider, and the full commit the runner was built from, with whether the working tree matched it. A version alone does not identify the runner, since two runs a commit apart report the same one. Where there is no commit, `runner_commit` is null and `runner_commit_absent_reason` says why: `no_checkout` for an installed wheel, `not_a_repository` for a source tree outside version control, and `vcs_unavailable` when git could not be run. `provider_kind` says whether the provider is built in or an external harness. `acvp-assay info --provider-command COMMAND` asks a harness to identify itself and reports what it declares instead, with the command beside it and without the `cryptography` and OpenSSL versions, which describe a library that did not answer. A run report's provider block carries the same `kind`, so a run through a harness cannot be read as a built-in one, and for a harness its `buildId`, or `null` with a `buildIdAbsentReason`, because a name and a version do not identify a build. The last command executes a tiny local fixture end to end and prints a JSON report.

New to the project? **[`docs/design.md`](docs/design.md) has the diagrams** — what the
system is for, how the two paths differ, and the end-to-end sequence for a run, a live
submission and the harness exchange.

## Running vectors

```bash
acvp-assay run VECTOR_FILE [--output RESULT_FILE] [--strict]
```

The algorithm is read from the vector file itself and routed automatically.

`VECTOR_FILE` is an ACVP-shaped `prompt.json`; an `expectedResults.json` must sit next to it in the same directory (every directory under `fixtures/` already follows this layout). Without `--output`, the JSON report is printed to stdout; with it, the report is written to `RESULT_FILE` instead. `--strict` also fails the run if any case is `SKIPPED` or `UNSUPPORTED`. See `docs/architecture.md` for the full exit-code table.

Every `UNSUPPORTED` case carries a `declineReason` beside its English `diagnostic`, from a closed set of four with four different repairs: `implementation_lacks` (the implementation under test lacks it, so the vendor fixes it), `runner_lacks` (this runner has not built it), `offline_undecidable` (no recorded answer can decide it, so only a submission to ACVTS can, or nothing can), and `vector_incomplete` (the vector lacks or contradicts what its own group requires). The summary counts each under `unsupportedByReason`, so one total never stands in for four different gaps, and splits them again under `unsupportedByClaimant` by who declined: what an external harness claimed about itself, which may only be `implementation_lacks` or `vector_incomplete`, and what this runner decided.

The summary also carries `concentration`: the two largest test groups' share of the cases, with the partition that produced it (`cases-by-tgId`) and how many groups there are. A case total reads as breadth and can be substantially one or two groups; NIST's pinned SHA2-256 set is 517 cases, and 99.8% of them are in two of its three groups.

## Coverage

Three questions a vendor actually needs answered, in one table:

- **Offline** — can this runner execute the family against an ACVP vector file?
- **Harness** — can *your* implementation answer it over `--provider-command`?
- **Live NIST verdict** — has NIST itself generated vectors for it, scored our answers, and
  said `passed`? The session ids are ours on `demo.acvts.nist.gov`; with your own credentials
  the same flow produces your own. Session state is gitignored, so nothing here is a claim you
  have to take on trust about *your* module — you run it yourself.

| Algorithm | Test types | Offline | Harness | Live NIST verdict |
| --- | --- | :---: | :---: | --- |
| `ACVP-AES-GCM` | AFT | ✅ | ✅ | `passed` — 765343 |
| `ACVP-AES-CCM` | AFT (encrypt and decrypt) | ✅ | ✅ | `passed` — 765788 |
| `ACVP-AES-ECB` | AFT, MCT | ✅ | ✅ | `passed` — 765342 |
| `ACVP-AES-CBC`, `-CTR`, `-OFB`, `-CFB128` | AFT, MCT, CTR | ✅ | ✅ | `passed` — 765353 |
| `ACVP-AES-KW`, `ACVP-AES-KWP` | AFT | ✅ | ✅ | `passed` — 765342 |
| `ACVP-AES-GMAC` | AFT | ✅ | ✅ | `passed` — 765518 |
| `ACVP-AES-XTS` | AFT (hex and number tweaks) | ✅ | ✅ | `passed` — 765786 |
| `ACVP-AES-CFB8` | AFT, MCT | ✅ | ✅ | `passed` — 766208 |
| `ACVP-AES-CFB1` | AFT, MCT (payload in bits) | ✅ | ✅ | `passed` — 766208 |
| `ACVP-AES-CBC-CS1`, `-CS2`, `-CS3` | AFT | ✅ | ✅ | `passed` — 766208 |
| `PBKDF` | AFT (all eleven approved HMACs) | ✅ | ✅ | `passed` — 766210 |
| `safePrimes` | keyVer; keyGen is generate-only ³ | ✅ | ✅ | `passed` — 766221 |
| `KAS-FFC-SSC` | AFT, VAL (dhEphem) | partial ³ | ✅ | `passed` — 766221 |
| `kdf-components` | AFT (`ssh` mode only ⁴) | ✅ | ✅ | `passed` — 766249 |
| `TLS-v1.2` | AFT (RFC 7627 extended master secret) | ✅ | ✅ | `passed` — 766249 |
| `TLS-v1.3` | AFT (DHE, PSK, PSK-DHE) | ✅ | ✅ | `passed` — 766249 |
| `KAS-IFC-SSC` | AFT, VAL (KAS1 and KAS2) | partial ³ | ✅ | `passed` — 766758 |
| `KTS-IFC` | AFT (KTS-OAEP-basic) | partial ³ | ✅ | `passed` — 766758 |
| `CMAC-AES` | AFT (gen and ver) | ✅ | ✅ | `passed` — 765342 |
| `SHA-1` | AFT, MCT | ✅ | ✅ | `passed` — 765345 |
| `SHA2-224` | AFT, MCT | ✅ | ✅ | `passed` — 765508 |
| `SHA2-256` | AFT, MCT | ✅ | ✅ | `passed` — 765339, 765342 |
| `SHA2-384`, `SHA2-512` | AFT, MCT | ✅ | ✅ | `passed` — 765508 |
| `SHA2-512/224`, `SHA2-512/256` | AFT, MCT | ✅ | ✅ | `passed` — 765508 |
| `SHA3-224`, `SHA3-384` | AFT, MCT | ✅ | ✅ | `passed` — 765508 |
| `SHA3-256`, `SHA3-512` | AFT, MCT | ✅ | ✅ | `passed` — 765345 |
| `SHAKE-128`, `SHAKE-256` | AFT | ✅ | ✅ | `passed` — 765794 |
| `HMAC-SHA-1` | AFT | ✅ | ✅ | `passed` — 765345 |
| `HMAC-SHA2-256` | AFT | ✅ | ✅ | `passed` — 765342 |
| `HMAC-SHA2-224/384/512`, `-512/224`, `-512/256` | AFT | ✅ | ✅ | `passed` — 765508 |
| `HMAC-SHA3-256` | AFT | ✅ | ✅ | `passed` — 765345 |
| `HMAC-SHA3-224`, `-384`, `-512` | AFT | ✅ | ✅ | `passed` — 765508 |
| `ctrDRBG` | AFT | ✅ | ✅ | `passed` — 765342 |
| `hashDRBG`, `hmacDRBG` | AFT | ✅ | ✅ | `passed` — 765354 |
| `KDF` (SP 800-108) | AFT | ✅ | ✅ | `passed` — 765343 |
| `KDA` (SP 800-56C, HKDF) | AFT, VAL | ✅ | ✅ | `passed` — 765811 |
| `ECDSA` | sigGen, sigVer | ✅ | ✅ | `passed` — 765343 |
| `KAS-ECC-SSC` | AFT, VAL (ephemeralUnified) | partial ³ | ✅ | `passed` — 765769 |
| `RSA` | sigGen, sigVer, signaturePrimitive, decryptionPrimitive | ✅ | ✅ | `passed` — 765356 |
| `ML-KEM` | encap, decap, key checks | harness only | ✅ | `passed` — 765724 ¹ |
| `ML-DSA` | sigVer (pure, external) | harness only | ✅ | `passed` — 765727 ¹ |

¹ Answered by the educational reference harness, not a shippable implementation — see the banner.

⁴ **`kdf-components` is one registry name covering nine component KDFs, and only `ssh` is built.**
It is the most common of them, on 46% of active FIPS 140-3 certificates. The others are declined
*by name* — a report says which mode is missing rather than marking the algorithm unsupported —
and `docs/algorithm-frequency.md` records where each sits: `ans9.63` 24%, `tls` 20%, `ikev2` 18%,
`ans9.42` 17%, `snmp` 17%, `srtp` 8%, `ikev1` 5%, `tpm` 0.1%.

**59 algorithm names across 34 families. All 59 reach a harness, all 59 can be submitted to a
live session, and all 59 have been.**

The harness path is checked against the built-in one by answering each pinned NIST prompt both
ways and comparing: **24,048 cases across ten families, byte-identical wherever the answer is
deterministic**. Where it cannot be, because the implementation invents part of the input, the
answers were checked for self-consistency instead — 10,950 KDF cases re-derived from the
`fixedData` the harness reported, and 800 signatures verified under the `qx`/`qy` it reported.
Groups the *reference* harness itself declines (`kwCipher: inverse`, SHAKE with ECDSA, TDES,
LDT) are excluded from that count rather than counted as passes, and PQC is not in it at all,
having no built-in side to compare against.

³ **Generated values cannot be checked offline, and are declined rather than guessed at.**
This covers KAS-ECC-SSC and KAS-FFC-SSC AFT cases, and `safePrimes` keyGen: each has the
implementation produce a fresh key, so the answer differs every run and no recorded value can
be compared with it. All are answered in full on submission, where the server recomputes them.
That is not a formality — session 766220's keyGen set failed on exactly this path while every
offline case was reported UNSUPPORTED, which is the honest outcome rather than a pass. An AFT
case has the implementation generate an ephemeral key pair, so Z differs on every run and cannot
be compared with the value NIST recorded from its own. The ACVP server *can* check it, because it
holds the peer private key and recomputes Z from the public key reported back — which is why the
live verdict above covers all 20 cases while `acvp-assay run` reports the 10 AFT ones UNSUPPORTED.
VAL cases supply every input and are fully checked offline.

### Per-family notes

| Family | Detail |
| --- | --- |
| SHA-1, SHA-2 | Both `standard` and `alternate` Monte Carlo chains; LDT is reported UNSUPPORTED |
| SHA-3 | Chains a single digest per iteration, not SHA-2's three |
| SHAKE | Revision FIPS202, AFT only. The output length is an input, so the same message squeezed to a different length is a different answer. The Monte Carlo chain is declined |
| HMAC | Honours per-group `macLen` truncation |
| AES chaining modes | Both directions of every Monte Carlo chain; CTR defines none. The IV advance differs per mode — see `docs/limitations.md` |
| AES-ECB | The 100 × 1000 chain, including the 192/256-bit key shuffle |
| AES-GMAC | Tag generation, and verification including deliberate forgeries |
| AES-CCM | 128/192/256-bit keys, 7 to 13-byte nonces, 32 to 128-bit tags. The tag is appended to the ciphertext rather than reported separately, so a zero-length payload still produces output |
| AES-KW/KWP | `kwCipher: cipher`; the `inverse` variant is reported UNSUPPORTED |
| AES-XTS | Revision 2.0, 128 and 256-bit keys. The key is two AES keys concatenated; a `number` tweak is a little-endian sequence number; a payload longer than `dataUnitLen` spans several units, each with its own tweak |
| RSA | PKCS#1 v1.5 and PSS over SHA-1/SHA-2/SHA-3; SHAKE masks reported UNSUPPORTED. `keyGen` is out of scope |
| ECDSA | P-224/256/384/521; sigVer is verdict-only, sigGen is verified against its own key |
| ctrDRBG | Both revisions; AES-128/192/256, with and without the derivation function. TDES reported UNSUPPORTED |
| hashDRBG, hmacDRBG | SHA-1, the SHA-2 family and both truncated SHA-512 variants |
| KDF SP 800-108 | Counter, feedback and double-pipeline modes over 14 PRFs. CMAC-TDES reported UNSUPPORTED |
| KDA SP 800-56C | HKDF mode, revisions Cr1 and Cr2. Only the `uPartyInfo||vPartyInfo` fixed-info pattern is assembled; others are declined rather than guessed at, since a wrongly built fixedInfo is wrong on every case |
| ML-KEM, ML-DSA | Require `--provider-command`: no built-in PQC implementation |

PQC has no built-in provider on purpose. `cryptography` 50.0.1 implements neither ML-KEM nor ML-DSA, and in a real engagement the implementation under test is the customer's — OpenSSL 3.5+, liboqs, an HSM, or their own module. `examples/pqc_reference_harness.py` drives the pinned NIST ML-KEM and ML-DSA sets end to end using `kyber-py` and `dilithium-py`, which are educational, **not constant-time**, and exist here to verify this runner and to demonstrate it, never as an implementation to ship or validate.

### Not covered

Named plainly so you can tell before you install whether this fits. None of these are
implemented, and none are silently mis-reported — an unrecognised algorithm exits with an
error rather than a pass:

`AES-XPN`, `AES-GCM-SIV` and the format-preserving modes (`FF1`, `FF3-1`); every TDES
family; `cSHAKE`, `KMAC`, `ParallelHash` and `TupleHash`; the key-agreement names beyond the
two SSC variants built here — `KAS-ECC`, `KAS-FFC`, `KAS-IFC` and `KAS-KC`; `DSA`, `EDDSA` and `DetECDSA`;
the eight `kdf-components` modes other than `ssh`;
`LMS` and `SLH-DSA`; `ConditioningComponent`; `Ascon`; `XECDH`; key
generation for RSA or ECDSA; and for the PQC names, everything but ML-KEM `encapDecap`
and ML-DSA `sigVer`.

How this list is chosen is no longer a matter of taste: `docs/algorithm-frequency.md`
counts every one of these across active FIPS 140-3 certificates, and
`docs/backlog.md` works down that ranking. Four of the names above appear on **no**
active certificate at all.

`acvp-assay run` on any of these reports the algorithm as unsupported and exits non-zero.

## How vendors use this

The built-in provider exercises OpenSSL through Python's `cryptography`, which is only useful
for checking the runner itself. **Testing *your* product means supplying it as a harness.**
Four stages, each independently useful — most vendors stop after stage 2:

### Stage 1 — see it work, no integration (5 minutes)

```bash
python3.12 scripts/dev.py setup
.venv/bin/python -m acvp_assay run fixtures/aes-gcm-valid-encrypt/prompt.json
```

Nothing of yours is involved yet. This confirms the tool runs and shows the report shape.

### Stage 2 — run NIST vectors against your implementation

Write a harness: a program that reads one JSON request per line on stdin and writes one JSON
response per line on stdout. That is the entire contract, so it can be a C binary talking to an
HSM, a shell script with `jq`, a Go service in front of a network appliance, or a Python script
driving a serial port. Nothing links against this project.

```bash
acvp-assay run vectors/SHA2-256-1.0/prompt.json \
    --provider-command "./my-harness --device /dev/hsm0" \
    --provider-timeout 30
```

Two worked starting points, neither importing anything from this package:

- **`examples/reference_harness.py`** — all 20 non-PQC operations in Python.
  (`examples/pqc_reference_harness.py` covers the four ML-KEM and ML-DSA operations
  separately, since it needs different dependencies.)
- **`examples/pkcs11/`** — a complete harness **in C, dispatching to a PKCS#11 token**, in
  one file with no dependencies beyond `libdl` and a PKCS#11 header. If your implementation
  is an HSM, start here: it answers AES-GCM, AES-ECB/CBC, the SHA-2 family including the
  Monte Carlo chain, and HMAC, verified through SoftHSM against 3,461 pinned NIST cases with
  no failures.

Implement only the families you are testing; decline the rest with `{"error": "unsupported"}`
and they are reported UNSUPPORTED rather than as failures.

You supply the vectors. Either use your own ACVTS-issued prompt files, or run
`python3 scripts/fetch_vectors.py` to pull the pinned upstream sets this project tests against.

### Stage 3 — submit your answers to NIST and let NIST judge

With ACVTS Demo credentials, the same harness answers vectors NIST generated for your session,
and NIST returns the verdict. See
[Submitting *your* implementation's answers](#submitting-your-implementations-answers).

```bash
python3 scripts/acvts_client.py submit --provider-command "./my-harness" --dry-run
```

### Stage 4 — keep it from breaking

`acvp-assay diff` compares two runs and fails CI on regressions, including coverage that
silently disappeared. See [Catching regressions between runs](#catching-regressions-between-runs).
With re-validation running well over a year, that is the failure mode that costs a cycle.

### The harness contract

**`docs/harness-protocol.md` is the full specification** — every operation, the reserved errors,
and worked integration patterns for HSMs, serial devices and network appliances.

```json
→ {"operation": "encrypt", "key": "000102…", "iv": "1011…", "aad": "", "pt": "4865…", "tagLen": 128}
← {"ct": "8C4B6FC3606396AE548B0DD4", "tag": "CEA4303CA9132112C1D14AE589AD15AF"}

→ {"operation": "decrypt", "key": "F0F1…", "iv": "A0A1…", "aad": "696E…", "ct": "8997…", "tag": "5333…"}
← {"pt": "646563727970742D6D65"}
← {"error": "authentication failed"}
```

```json
→ {"operation": "encrypt", "key": "000102…", "iv": "1011…", "aad": "", "pt": "4865…", "tagLen": 128}
← {"ct": "8C4B6FC3606396AE548B0DD4", "tag": "CEA4303CA9132112C1D14AE589AD15AF"}

→ {"operation": "decrypt", "key": "F0F1…", "iv": "A0A1…", "aad": "696E…", "ct": "8997…", "tag": "5333…"}
← {"pt": "646563727970742D6D65"}
← {"error": "authentication failed"}
```

The 24 operations, by family — implement only the rows you are testing:

| Family | Operations |
| --- | --- |
| Always | `metadata` |
| AES-GCM | `encrypt`, `decrypt` |
| Hashes | `digest`, `digest-mct` |
| HMAC | `mac` |
| AES block modes, ECB | `block-transform`, `block-mct` |
| CMAC, GMAC, KW/KWP | `cmac`, `gmac`, `key-wrap` |
| ECDSA | `ecdsa-sign`, `ecdsa-verify`, `ecdsa-sign-group` |
| RSA | `rsa-sign-group`, `rsa-verify`, `rsa-primitive-sign`, `rsa-primitive-decrypt` |
| DRBGs | `drbg` |
| KDF SP 800-108 | `kdf-108` |
| ML-KEM, ML-DSA | `ml-kem-encapsulate`, `ml-kem-decapsulate`, `ml-kem-key-check`, `ml-dsa-verify` |

`ecdsa-sign-group` and `rsa-sign-group` are needed only if you intend to *submit* sigGen to a
live session: ACVP reports the public key once per group, so every case in a group must share
one key, which per-case signing cannot express.

Two error values are reserved. `{"error": "unsupported"}` declines a case the implementation does not offer — a curve, a parameter set, a mode — and is reported UNSUPPORTED rather than as a failure, because capability is yours to declare, not ours to assume. (An HSM's binary curves are not "unsupported" merely because Python's `cryptography` lacks them.)

A rejected authentication tag is reported as `{"error": "authentication failed"}`, **not** as a crash or a non-zero exit. This matters: roughly a third of NIST's own AES-GCM decrypt cases are deliberate failures where rejecting the tag is the correct answer, and a harness that dies on them will score a conforming implementation as broken.

The Monte Carlo chain is delegated whole via `digest-mct`: at 100,000 inner iterations, one round trip per hash would take hours, and running the chain is what a real implementation under test does anyway.

`examples/reference_harness.py` is a complete worked implementation that imports nothing from this package. `--provider-timeout SECONDS` bounds each call, so a wedged device cannot hang the run. Its stderr passes through to your terminal for debugging but never enters the JSON report, since a crashing harness may print key material.

**The harness is started once and kept alive** for the whole run, so expensive
setup — a PKCS#11 login, a serial port, an SSH session — happens once rather
than once per case. That is worth about fifty times the run time: 239 SHA3-256
cases take 0.5 s against a persistent harness and 17.9 s against one spawned per
case. A one-shot harness that reads stdin to end still works and is detected
automatically, because a shell script with `jq` naturally takes that shape and
reach matters more than the speed lost.

### Sample output — PASS

```bash
$ acvp-assay run fixtures/aes-gcm-valid-encrypt/prompt.json; echo "exit: $?"
```

```json
{
  "cases": [
    {
      "actual": { "ct": "8C4B6FC3606396AE548B0DD4", "tag": "CEA4303CA9132112C1D14AE589AD15AF" },
      "expected": { "ct": "8C4B6FC3606396AE548B0DD4", "tag": "CEA4303CA9132112C1D14AE589AD15AF" },
      "status": "PASS",
      "tcId": 1,
      "tgId": 1
    }
  ],
  "provider": { "name": "cryptography-aes-gcm", "...": "..." },
  "summary": { "total": 1, "passed": 1, "failed": 0, "errored": 0, "skipped": 0, "unsupported": 0,
               "unsupportedByReason": { "implementation_lacks": 0, "offline_undecidable": 0, "runner_lacks": 0, "vector_incomplete": 0 },
               "unsupportedByClaimant": { "harness": { "implementation_lacks": 0, "vector_incomplete": 0 }, "runner": { "implementation_lacks": 0, "offline_undecidable": 0, "runner_lacks": 0, "vector_incomplete": 0 } },
               "concentration": { "partition": "cases-by-tgId", "cardinality": 1, "largestTwoShare": 1.0 } }
}
```
```text
exit: 0
```

### Sample output — failure

```bash
$ acvp-assay run fixtures/aes-gcm-invalid-decrypt-tag/prompt.json; echo "exit: $?"
```

```json
{
  "cases": [
    {
      "actual": null,
      "diagnostic": "authentication failed",
      "expected": {},
      "status": "ERROR",
      "tcId": 1,
      "tgId": 1
    }
  ],
  "provider": { "name": "cryptography-aes-gcm", "...": "..." },
  "summary": { "total": 1, "passed": 0, "failed": 0, "errored": 1, "skipped": 0, "unsupported": 0,
               "unsupportedByReason": { "implementation_lacks": 0, "offline_undecidable": 0, "runner_lacks": 0, "vector_incomplete": 0 },
               "unsupportedByClaimant": { "harness": { "implementation_lacks": 0, "vector_incomplete": 0 }, "runner": { "implementation_lacks": 0, "offline_undecidable": 0, "runner_lacks": 0, "vector_incomplete": 0 } },
               "concentration": { "partition": "cases-by-tgId", "cardinality": 1, "largestTwoShare": 1.0 } }
}
```
```text
exit: 1
```

This second fixture's tag is deliberately corrupted (see `fixtures/README.md`); it exists to prove the tool surfaces a real, deterministic failure instead of swallowing it.

## Verified against NIST's own server

Static vector files tell you whether a runner agrees with a snapshot. They cannot tell you whether
it agrees with the system that issues the vectors. So all 59 supported algorithm names have been
through a live test session on NIST's ACVTS Demo server:
register capabilities, fetch vectors NIST generated for this client, compute answers, submit them,
and read back the verdict.

Two qualifications, both of which a reader should have without asking. These were **sample**
sessions, so NIST supplies the expected results alongside the prompt — but the answers submitted
were computed from the prompt by the same providers the offline runner uses, never read out of
NIST's answer key; `responder.py` does not open `expectedResults.json` at all. And Demo is not the
production ACVTS, which is available to accredited laboratories rather than to tool authors.

| Session | Algorithms | Sets | Cases | Verdict |
| ---: | --- | ---: | ---: | --- |
| 765339 | SHA2-256 | 1 | 513 | `passed` |
| 765342 | SHA2-256, HMAC-SHA2-256, AES-ECB, CMAC-AES, AES-KW, AES-KWP, ctrDRBG | 7 | 8,966 | `passed` |
| 765343 | AES-GCM, KDF SP 800-108, ECDSA sigGen, ECDSA sigVer | 4 | 647 | `passed` |
| 765345 | SHA-1, SHA3-256, SHA3-512, HMAC-SHA-1, HMAC-SHA3-256 | 5 | 2,877 | `passed` |
| 765346 | AES-CBC, AES-CTR, AES-OFB, AES-CFB128 | 4 | — | **abandoned**, see below |
| 765353 | AES-CBC, AES-CTR, AES-OFB, AES-CFB128 | 4 | 6,016 | `passed` |
| 765354 | hashDRBG, hmacDRBG | 2 | 120 | `passed` |
| 765356 | RSA sigGen, sigVer, signaturePrimitive, decryptionPrimitive | 4 | 98 | `passed` |
| 765508 | SHA2-224/384/512, SHA2-512/224, SHA2-512/256, SHA3-224, SHA3-384, and the eight remaining HMACs | 15 | 12,867 | `passed` ¹ |
| 765518 | AES-GMAC | 1 | 360 | `passed` |
| 765724 | ML-KEM encapDecap | 1 | 165 | `passed` ² |
| 765727 | ML-DSA sigVer | 1 | 45 | `passed` ² |
| 765769 | KAS-ECC-SSC | 1 | 20 | `passed` |
| 765786 | ACVP-AES-XTS | 1 | 480 | `passed` |
| 765788 | ACVP-AES-CCM | 1 | 4,830 | `passed` |
| 765794 | SHAKE-128, SHAKE-256 | 2 | 508 | `passed` |
| 765811 | KDA (HKDF) | 1 | 300 | `passed` |
| 766207 | AES-CFB8, AES-CFB1, AES-CBC-CS1/2/3 | 5 | — | **CFB1 failed**, see below |
| 766208 | AES-CFB8, AES-CFB1, AES-CBC-CS1/2/3 | 5 | 17,656 | `passed` |
| 766210 | PBKDF | 1 | 110 | `passed` |
| 766220 | safePrimes, KAS-FFC-SSC | 3 | — | **keyGen failed**, see below |
| 766221 | safePrimes, KAS-FFC-SSC | 3 | 98 | `passed` |
| 766249 | kdf-components (ssh), TLS-v1.2, TLS-v1.3 | 3 | 410 | `passed` |
| 766758 | KAS-IFC-SSC, KTS-IFC | 2 | 30 | `passed` |
| | **Completed** | **65** | **57,116** | **all `passed`** |

² Answered through `examples/pqc_reference_harness.py`. `cryptography` implements neither
ML-KEM nor ML-DSA, so there is nothing built in to answer with; the harness is backed by
`kyber-py` and `dilithium-py`, which are educational and not constant-time. These two verdicts
say the runner handles PQC vector sets correctly, and say nothing about any shippable
implementation. Both sessions registered narrowly — ML-DSA `pure`/`external` only — because
`preHash` and `externalMu` groups are refused by design rather than answered.

¹ Session 765508 registered a sixteenth algorithm, AES-GMAC, whose vector set NIST's generator
refused with `min must be less than max` — the registration declared a zero-width `payloadLen`,
and GMAC has no payload to describe. That is a bug in the capability file, not in the runner: no
vectors were ever produced, so there was nothing to answer. The set is excluded from the completed
total, the session therefore reports `passed: false` overall, and GMAC was re-registered correctly as 765518.

**Session 765346 is listed because it failed.** It is where the Monte Carlo decrypt bug below
was found: the AES-CTR set was submitted, the other three were not, and the session was
abandoned rather than finished around a known-wrong answer. 765353 is the re-run after the fix.
Its cases are excluded from the total — a run that was abandoned is not evidence, and dropping
it from the table without saying so would make the total flattering rather than true.

**On evidence.** `acvts_client.py results` now writes the server's reply to `results.json` beside
the vector sets it judges, so every session from 765508 onward — sixteen of them — is backed
by a verdict still on disk. The seven earlier ones are not: `results` used to print the reply and
discard it, and ACVP scopes a session's token to its registration, so re-reading them now returns
403. Those seven verdicts are recorded here from the runs themselves and cannot be re-fetched —
which is exactly why every session since is written down.

### What this caught that fixtures did not

Each of these passed the offline suite and would have shipped:

- **The AES-GCM parser rejected every real vector set.** It required `ivGenMode`, which qualifies
  *internal* IV construction — so the live server omits it whenever `ivGen` is `external`. The
  pinned upstream sample file happens to include it, which is exactly why this survived.
- **The chaining-mode Monte Carlo chains were wrong for decryption.** The specification writes the
  inner loop as a cipher that "continues" from the previous call without saying what that does to
  the IV. Read literally it reproduces the encrypt arrays exactly and disagrees from the first
  block when decrypting. The rules came from NIST's generator, not the prose.
- **RSA-PSS ignored `maskFunction`.** FIPS 186-5 lets PSS use SHAKE as its mask generation
  function, signalled by a field separate from `hashAlg`. Six sigVer cases failed while their
  group looked perfectly supported.
- **ECDSA and RSA sigGen generated a fresh key per case.** ACVP reports the public key once per
  *group*, so a key per case cannot be expressed in the response document at all.
- **safePrimes generated private keys outside the subgroup range.** The key had to come from
  [1, q−1] where q = (p−1)/2; it came from [1, p−2]. This is the subtlest defect the project
  has found, because the generator 2 has order q, so g^x = g^(x mod q) — an out-of-range x
  produces a **perfectly valid public key**. The obvious check, generating a pair and then
  verifying it, passed every time: `key_ver` was right, `key_gen` was wrong, and the two
  agreed with each other. Nor could any vector catch it — all 60 of NIST's keyVer cases use a
  key already inside the range, so that mode never exercises the constraint at all. It is also
  precisely the case this runner *declines by design*: keyGen produces a fresh key, so the
  offline run reported all 18 cases UNSUPPORTED with the reason "submit to ACVTS, which
  recomputes it". Session 766220 did, and ACVTS said no. Where CFB1 was two code paths with
  one unchecked, this was a single path, wholly self-consistent, and still wrong.
- **The response builder dropped CFB1's bit count.** CFB1 is the only mode whose payload is
  measured in bits rather than bytes, and the hex encoding pads the rest of the byte. The
  offline runner passes the declared `payloadLen`, so every case agreed with NIST's own sample
  file and passed. The submission path builds its document through a different function, that
  function never passed the bit count, and so it answered over the padding as well. Session
  766207 returned `fail` for CFB1 while the offline run was green on all 2,144 cases;
  766208 is the same five names after the fix. It is the clearest example in the project of a
  defect that only the live server can find: the two paths were checked against the same file
  and only one of them was checked by NIST.

### Reproducing it

Credentials are yours to obtain — write to `acvts-demo@nist.gov` — and never live in this
repository. The client reads them from the environment:

```bash
export ACVTS_CERT=/path/to/your.cer ACVTS_KEY=/path/to/your.key ACVTS_SEED=/path/to/totp.txt

python3 scripts/acvts_client.py check                              # credentials, no network
python3 scripts/acvts_client.py register acvts-capabilities/drbg.json
python3 scripts/acvts_client.py fetch                              # prompts, and expected results
acvp-assay run .acvts/session-765354/4032194/prompt.json           # verify offline
python3 scripts/acvts_client.py submit                             # let NIST judge
python3 scripts/acvts_client.py results
```

### Submitting *your* implementation's answers

Without `--provider-command` the built-in providers answer, which tests this runner rather
than your product. Point it at your harness and every value NIST scores comes from your code:

```bash
python3 scripts/acvts_client.py submit \
    --provider-command "./my-acvp-harness --device /dev/hsm0" \
    --dry-run                                       # writes response.json, sends nothing
```

Drop `--dry-run` once the documents look right. Two behaviours differ from an offline run,
both deliberate:

- **A case your harness declines refuses the whole submission.** Offline, UNSUPPORTED is a
  useful verdict. Here there is no such verdict — ACVP scores a missing case as a *wrong
  answer* — so a partial document would record a failure you never earned. The error names
  the operation that was declined, so you can either implement it or narrow your
  registration.
- **Capability is yours to declare.** Modes the built-in provider lacks, such as DRBG `TDES`
  or KDF `CMAC-TDES`, are sent to your harness rather than refused on your behalf.

Session state, downloaded vectors and tokens land in a gitignored `.acvts/`. NIST's vectors are
theirs to distribute, and the certificate, key and TOTP seed are secrets.

One detail the public documentation gets wrong, in case it saves you an afternoon: the TOTP is
**HMAC-SHA-256 with eight digits**, not the SHA-1 and six digits the ACVP wiki and issue #297
imply. A wrong guess returns a bare 401 naming neither factor.

## Catching regressions between runs

Getting a certificate is one question; *staying* conformant is another, and with re-validation
running well over a year a silent break can survive to the next cycle. Compare two reports:

```bash
acvp-assay run vectors/ACVP-AES-GCM-1.0/prompt.json --output baseline.json
# ... upgrade a library, change firmware, bump a container base image ...
acvp-assay run vectors/ACVP-AES-GCM-1.0/prompt.json --output current.json

acvp-assay diff baseline.json current.json
```

```text
verdict: REGRESSED
provider changed between runs:
  baseline: cryptography-aes-gcm, cryptography 50.0.1, OpenSSL OpenSSL 3.5.0 8 Apr 2025
  current : cryptography-aes-gcm, cryptography 50.0.1, OpenSSL OpenSSL 4.0.2 25 Aug 2026
regressed: 1
  tgId 1 tcId 1: PASS -> FAIL (tag mismatch)
coverage lost: 10
  tgId 2 tcId 16: PASS -> UNSUPPORTED [runner_lacks] (ivGen 'internal' is not supported)
  ... and 9 more
```

Exit codes: 0 when nothing got worse, 1 on a regression, 2 when a report cannot be read — so
`acvp-assay diff` drops straight into CI. `--output` writes the machine-readable diff.

**Coverage loss counts as a regression.** A case that used to run and is now `UNSUPPORTED`,
`SKIPPED`, or simply absent is reported as loudly as an outright failure, because that is the
failure mode that hides: the totals still look clean, since the case has stopped being counted.
Provider identity is diffed alongside the cases, since a changed library or backend is usually the
cause rather than a detail.

A case that stays `UNSUPPORTED` for a different reason is listed as well, under `decline reason
changed`, though it is neither a regression nor a fix. A gap that moved from `runner_lacks` to
`implementation_lacks` has a different owner, and the totals cannot show it.

## Development commands

```bash
# Install the package and development dependencies into .venv
python3.12 scripts/dev.py setup

# Run formatting checks, lint, static typing, and tests
python3.12 scripts/dev.py test

# Run all checks and build wheel/sdist artifacts
python3.12 scripts/dev.py verify

# Print provider/runtime metadata
python3.12 scripts/dev.py demo
```

## Repository map

- `src/acvp_assay/`: application package
- `scripts/dev.py`: the setup, test and verify gate
- `scripts/acvts_client.py`: live ACVTS client — register, fetch, submit, results
- `scripts/fetch_vectors.py`: downloads and hash-verifies the pinned upstream vectors
- `scripts/cavp_frequency.py`: counts ACVP algorithm names across every active FIPS 140-3
  certificate, which is how the build order is chosen
- `acvts-capabilities/`: capability registrations used for the live sessions
- `examples/`: worked reference harnesses that import nothing from the package —
  Python, and a PKCS#11 one in C under `examples/pkcs11/`
- `tests/unit/`: focused unit tests
- `tests/integration/`: subprocess and full-path tests
- `fixtures/`: small, rights-safe local test vectors
- `docs/design.md`: **HLD and LLD diagrams** — system context, layers, and end-to-end
  sequences for verifying, answering a live session, and the harness exchange
- `docs/architecture.md`: component boundaries and data flow
- `docs/harness-protocol.md`: the full harness specification for vendors
- `docs/limitations.md`: security and assurance boundaries
- `docs/vector-sources.md`: pinned upstream source, hashes, licensing, and redistribution policy
- `docs/algorithm-frequency.md`: **what FIPS 140-3 certificates actually contain** — every
  algorithm name counted across 690 active certificates, and what that changed
- `docs/decisions/`: committed design decisions
- `docs/backlog.md`: what is built and what is next
- `SERVICES.md`: paid engagements — readiness assessment, harness build, regression retainer
- `CONTRIBUTING.md`: what a change is reviewed against, stated before a pull request rather than in it
- `CHANGELOG.md`: release history
- `BUILDLOG.md`: running record of how each family was built and verified

## Safety

Do not add credentials, employer code or data, proprietary vectors, or confidential screenshots. Use only vectors whose source and redistribution terms have been recorded.

## License

MIT; see `LICENSE`.
