Metadata-Version: 2.5
Name: vulntrack
Version: 0.4.1
Summary: Assess Harbor/Trivy findings with auditable, ordered CVSS environmental rules (append-only parquet store)
Author-email: brunnelu <6707792+brunnelu@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: cvss>=3.6
Requires-Dist: httpx>=0.27
Requires-Dist: polars>=1.17
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pydantic>=2.7
Requires-Dist: tenacity>=9.1.4
Requires-Dist: textual>=8.2.8
Requires-Dist: tomli>=2.0; python_version < '3.11'
Description-Content-Type: text/markdown

# vulntrack

Assess Harbor/Trivy findings with an **auditable rules engine**. A rule reads columns
and sets columns; every rule that matches applies, and every assessment carries the
sentences that produced it. Rules and packs are TSVs in git. Persistence is
**append-only parquet in blob storage** (an Azure container, or a local path).

| | |
|---|---|
| how rules work | [docs/rules.md](docs/rules.md) |
| how packs work | [docs/packs.md](docs/packs.md) |
| architecture, storage, collection | [docs/index.md](docs/index.md) |
| packaging, versions, releases | [docs/packaging.md](docs/packaging.md) |
| what changed, what is still open | [WORKLOG.md](WORKLOG.md) |

This file is how to run it. It deliberately does not explain the model — that is
`docs/rules.md` and `docs/packs.md`, each of which is the only description of its
subject.

## Quickstart

```bash
uv sync

uv run vulntrack setup            # print the sample config, with comments
uv run vulntrack setup --write    # create vulntrack.toml, .env, rules/, packs/ and tools/

uv run vulntrack collect --no-submit        # progress bar on a terminal; --no-progress off

# -v logs every HTTP request (httpx's own line)
uv run vulntrack -v collect --no-submit

# map extra columns onto the frame — deployments, KEV, owners, anything
uv run vulntrack packs list      # pack order, inferred keys, the dependency graph
uv run vulntrack packs check     # duplicates, collisions, coverage, dead rules

# rules are tables: one row is one reason. See rules/README.md
uv run vulntrack rules list
uv run vulntrack rules add --id cve.x --when 'cve=^CVE-2024-0001$' \
  --set-metric MA=N --because 'Why this does not apply here.'

# how the estate is being rated, and which .tsv row to add next
uv run vulntrack progress

# record the current assessments so changes over time are queryable
uv run vulntrack snapshot --dry-run
uv run vulntrack snapshot

# Textual TUI — walk residue by finding PK, apply rules (recommended)
# scope a rule by ticking columns (cve on by default; any column a pack maps is
# offered), watch what it covers, and see the images it lands on — `i` toggles them
uv run vulntrack ui

# same flow without TUI
uv run vulntrack assess
uv run vulntrack assess CVE-2024-0001 --set MA=N --because "…"
```

### What `collect` shows while it runs

Four phases on stderr, one rewritten line each, with elapsed time and an estimate:

```
repos   512 repositories found  0:11
images  [━━━━━━━━━━━             ] 235/512  4:11 eta 4:56  1893 artifacts
reports [━━━━━━━                 ] 820/2803  6:42 eta 16:12  3.8M findings · library/proxy
reports [━━━━━━━                 ] 820/2803  6:44 eta 16:12  writing 50k findings
scans   [━━━━━                   ] 47/210  0:31 eta 2:15  restart · library/proxy
```

Every phase counts a different noun, and in two of them the count is not what takes
the time — so each line carries a tally of the thing that does.

`images` counts **repositories, not images** — one repository holding three hundred
artifacts is a single step, which is why the count can sit still for a while. The
artifact tally beside it is what moves in between, one page of a hundred at a time,
so a run that is working never looks like a run that has hung.

`reports` counts **artifacts, not findings**, and the two do not track each other: a
base image with a thousand findings and a scratch image with two are one step each.
The findings tally is the volume. Every `batch_rows` findings the run stops reading
and writes a slice, which on a large store is the slowest single step in the phase
and finishes no report while it runs — that is the `writing` line.

The estimate is the plainest one that can be checked by hand: elapsed divided by
what is finished, times what is left. It says `eta --:--` rather than guessing when
it has no total (`repos` has none until it ends) or no pace yet. Both repositories
and reports differ wildly in size, so it moves around early and settles late.

`--no-progress` turns the whole thing off; it is already off when stderr is not a
terminal, so piping to a file leaves no escape codes behind.

### The log file

The progress line is for watching; the log file is for afterwards.

```
vulntrack --log-file vulntrack.log collect
VULNTRACK_LOG_FILE=vulntrack.log vulntrack collect     # same thing, for cron
```

stderr stays at WARNING so it does not fight the progress bar. The file takes INFO
always — every store object written, each phase's counts, every scan Harbor refused
— and `-v` adds httpx's line per request.

Every line carries `run=<id>`, and that is the same id the store stamps into every
row and into every object filename:

```
2026-09-08 14:50:06 INFO  run=15f6c18275a0 vulntrack.store wrote findings rows=50000
                          …/findings/dt=2026-09-08/145006-15f6c18275a0-0001.parquet
```

So an object you are suspicious of names the run that wrote it, and the log says what
that run was doing. `--watch` puts many runs in one process, which is why the id is on
every line rather than in a banner at the top.

### Artifacts Harbor will not scan

A submit that comes back 4xx — an image whose media type the scanner does not handle
answers `400` — is that artifact's problem, not the run's. It is counted as
`scans_failed`, logged with Harbor's own message, and the run carries on submitting
the rest. `5xx` still stops the run: that is a fact about the registry, and the next
submission would fail the same way.

An expired credential is a 4xx too and is counted rather than raised. That is safe to
read, because it fails *every* artifact: `scans_failed` comes back equal to the total,
which no ordinary run produces.

## Versions and releases

**There is no version number written anywhere in this repository.** It is derived
from git, so the only way to declare one is to tag a commit — which is also the
only way to make a release.

```bash
vulntrack --version
0.3.0                              # built from tag v0.3.0
0.3.0.dev3+g1d2f73a.d20260908      # three commits past it, built on the 8th
```

A dev build numbers itself: one increment above the last release, below the next,
carrying the commit and the date. Nothing to bump, nothing to forget, and two
builds of different commits can never claim the same number.

One caveat in a working copy: the version is stamped when the package is
*installed*, and `importlib.metadata` reads it back, so in an editable checkout
`--version` names the commit you last synced at rather than `HEAD`. Run `uv sync`
to restamp it. A built wheel is always exact, which is the case that matters.

The `+g<commit>.d<date>` part is load-bearing — **PyPI refuses any version that
has one** — so an untagged build cannot reach PyPI whatever a workflow is told to
do with it. Releasing is therefore just: tag, then publish a GitHub Release. That
uploads to PyPI and attaches the wheel, the sdist and the base64 below to the
release itself.

```bash
sh scripts/check_publishable.sh 0.3.0 v0.3.0   # the same check CI runs
```

Full procedure, and the one-time Trusted Publishing setup: **[docs/packaging.md](docs/packaging.md)**.

## Install where PyPI is unreachable — the wheel as base64

A wheel is a zip file, and plenty of places will not carry one: a chat window, a
ticket, a jump host you reach with nothing but a terminal. Base64 makes it text
you can paste, for about a third more bytes.

**Produce it.** Locally, or in CI with the **Wheel as base64** workflow (Actions →
run it; it builds any commit, publishes nothing, and puts the digest and these
same commands on the run summary). Both call one script, so a wheel handed over
from CI and one you encoded by hand are byte-identical:

```bash
sh scripts/wheel_base64.sh --build     # uv build, then encode
sh scripts/wheel_base64.sh             # encode whatever is already in dist/
```

It writes `dist/*.whl.b64` next to the wheel and prints what the far side needs:

```
wheel    dist/vulntrack-0.2.0b3-py3-none-any.whl  (93050 bytes)
base64   dist/vulntrack-0.2.0b3-py3-none-any.whl.b64  (125701 bytes, 1633 lines)
sha256   ffe776f1d2674e8b8f958c8d05fc8a45b8c898eb1abc8eaddf5459b0a9649067
```

**Install it.** Paste the `.b64` into a file on the far side, then:

```bash
base64 -d vulntrack-0.2.0b3-py3-none-any.whl.b64 > vulntrack-0.2.0b3-py3-none-any.whl
echo 'ffe776f1…  vulntrack-0.2.0b3-py3-none-any.whl' | sha256sum -c -
pip install ./vulntrack-0.2.0b3-py3-none-any.whl
```

**Check the digest — it is not decoration.** A paste either survives intact or
truncates, and a truncated `.b64` decodes to a truncated zip that pip may install
as a half-package while reporting success. `sha256sum -c` is what turns a bad
paste into a failure you can see, before you are debugging a missing module.

Two portability notes: `base64 -d` is GNU, macOS wants `base64 -D`, and
`python -c 'import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.read()))'`
works everywhere. The wheel is pure Python and needs 3.10 or newer — but `pip
install` still resolves vulntrack's dependencies from an index, so on a machine
with no PyPI at all you need those wheels too (`pip download` them alongside, or
point pip at an internal mirror).

## Blob storage & Azure credentials

`store` can be a local path or `az://container/prefix`. Credentials are **injected,
never constructed inside the store** — whoever runs vulntrack owns the auth decision.
Three ways in, **commonest first**. Pass more than one and `open_store` takes the
*provider* ahead of the credential object, and either ahead of the config — the order
is in [the architecture](docs/index.md#7-storage), which is where it is settled.

**1. A credential object you already hold** (the injection point for an app that has
already authenticated):

```python
from azure.identity import DefaultAzureCredential
from vulntrack.settings import Settings
from vulntrack.store import open_store

store = open_store(Settings(), credential=DefaultAzureCredential())
findings, images = store.findings(), store.images()
```

Any object with a `get_token(*scopes)` returning `.token` / `.expires_on` works —
`ManagedIdentityCredential`, `AzureCliCredential`, `ClientSecretCredential`, or your
own wrapper. It is exchanged for a bearer token that polars refreshes on expiry, so a
long `collect` does not die an hour in.

**2. A provider callable**, if you want full control of caching or a non-Azure token:

```python
store = open_store(Settings(), credential_provider=lambda: ({"bearer_token": tok}, exp))
```

The contract is polars': return `({key: value}, expiry_epoch_or_None)`.

**3. A name in `vulntrack.toml`** — the convenience path the CLI uses. Needs the
`azure-identity` package; nothing secret goes in the file:

```toml
store = "az://vulntrack/prod"
azure_credential = "default"   # default | cli | managed_identity | environment | workload_identity
```

For an account key or SAS instead, hand object_store its options directly. These *are*
secrets, so prefer the environment (`VULNTRACK_STORAGE_OPTIONS`) over the file:

```toml
[storage_options]
account_name = "myaccount"
```

`storage_options` is merged in either way, so the account name can live in the config
while the token arrives from code. A local `store` path ignores all of it.

**Order matters, and TOML will not warn you.** `[storage_options]` is a table, so every
key below it belongs to it, to the end of the file — put it *after* every plain
`key = value`, or `host`, `concurrency` and the rest silently move inside it. The
shipped template keeps all its tables at the bottom for this reason.

The container is the first segment of `store` (`az://CONTAINER/PREFIX`); the account is
`account_name`. Two settings, read together — `vulntrack images` is the cheapest way to
find out you got one of them wrong.

> Untested against a real Azure account — there is no tenant available here. The
> contract is pinned by tests using a fake credential (`tests/test_store_credentials.py`),
> including the `bearer_token` key name, which was taken from polars'
> `CredentialProviderAzure` rather than guessed.

## Layout

| path | role |
|---|---|
| `vulntrack/engine/` | frame + packs, rules as tables, CVSS environmental score |
| `vulntrack.toml` | every setting (gitignored; written by `vulntrack setup --write`) |
| `packs/*.tsv` | mapping packs joined onto the frame (templates in `vulntrack/templates/`) |
| `rules/*.tsv` | rules tables — one row is one reason (template `vulntrack/templates/rules/`) |
| `.env` | every secret — `HARBOR_<PROJECT>_PASSWORD` |
| `vulntrack/settings.py` | env → `.env` → `vulntrack.toml` → defaults |
| `vulntrack/templates/` | what `setup` copies: config, `.env`, packs, seed ruleset |
| `vulntrack/store.py` | append-only parquet store, SCD2 views |
| `vulntrack/harbor.py` | Harbor client + collect job |
| `vulntrack/engine/scope.py` | what a rule applies to: the toggle list, and what ticking one means |
| `vulntrack/tui.py` | Textual assess UI |
| `scripts/wheel_base64.sh` | the wheel as pasteable text, for an air-gapped install |
| `scripts/check_publishable.sh` | can this version go to PyPI? the same check CI runs |
| `vt-store/` | runtime data (gitignored) |

### What code runs when

One path per command, so "which code produced this number" is answerable without
reading the whole tree. Read top to bottom; every rating goes through the same four
steps in the same order.

```text
vulntrack collect     harbor.py            Harbor API  → store.append()      (writes)
                      store.py             append-only parquet, change-detected

vulntrack rate        store.py             load findings + images
  │                   engine/frame.py      build_frame()  ─ one row per occurrence
  │                   engine/packs.py      make_packs() → apply_pack()  ─ LEFT join, row count fixed
  │                   engine/rules.py      Reason.mask()  ─ one polars expr per condition
  │                   engine/rating.py     _metrics()  ─ min over the matching rules
  └─►                 engine/rating.py     _score()    ─ scored per distinct vector

vulntrack assess      engine/rating.py     residue() over the same rated frame
vulntrack ui          tui.py               Textual; the same rate() call
  │                   engine/scope.py      offer()  ─ scopeable columns, derived from the packs
  └─►                 engine/rules.py      Reason.mask() again ─ the coverage preview
vulntrack packs …     engine/packs.py      coverage(), redundant_key_columns(), unproduced_rule_columns()
vulntrack explain …   engine/packs.py      explain_column() walks the trail backwards
vulntrack snapshot    store.py             append() — change-detected, PK-keyed
vulntrack progress    cli.py               the same rate() call, summarised
vulntrack setup       cli.py               copies *.example — no logic, on purpose
```

Two properties worth knowing, because they are what keeps the above auditable:

- **`Reason.mask()` is the only place a condition is evaluated.** Rating and the
  coverage preview call the same expression, so the preview cannot disagree with the
  rating. There is no second row-at-a-time implementation. `Store.pending()` is the
  same idea for writes: `snapshot --dry-run` asks the writer what it would do rather
  than re-deriving the diff.
- **Reasons are applied in order, and the later one wins.** Precedence is a stored
  number per rule, not a line position — see [docs/rules.md](docs/rules.md).
- **`apply_pack` asserts the row count is unchanged.** Every join is a left join and
  the frame is exactly the findings Harbor reported — see `docs/packs.md`, "Invariant:
  every join is a left join".
