Metadata-Version: 2.5
Name: wana
Version: 0.5.0
Summary: Score, select and audit fine-tuning datasets with bundled offline model weights.
Project-URL: Homepage, https://github.com/sauloleite/wana
Project-URL: Issues, https://github.com/sauloleite/wana/issues
Author-email: Saulo Leite <sjoldeveloper@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: contamination,dataset,fine-tuning,jsonl,provenance,sft
Classifier: Development Status :: 3 - Alpha
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: llama-cpp-python<0.4,>=0.3.7
Requires-Dist: llm-smollm2==0.1.2
Provides-Extra: all
Requires-Dist: numpy>=1.26; extra == 'all'
Requires-Dist: onnxruntime>=1.20; extra == 'all'
Requires-Dist: tiktoken>=0.7; extra == 'all'
Requires-Dist: tokenizers>=0.20; extra == 'all'
Requires-Dist: torch>=2.4; extra == 'all'
Requires-Dist: transformers<6,>=4.46; extra == 'all'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: hypothesis>=6; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pre-commit>=4; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine>=6; extra == 'dev'
Provides-Extra: embed
Requires-Dist: numpy>=1.26; extra == 'embed'
Requires-Dist: onnxruntime>=1.20; extra == 'embed'
Requires-Dist: tokenizers>=0.20; extra == 'embed'
Provides-Extra: score
Requires-Dist: torch>=2.4; extra == 'score'
Requires-Dist: transformers<6,>=4.46; extra == 'score'
Provides-Extra: tokens
Requires-Dist: tiktoken>=0.7; extra == 'tokens'
Description-Content-Type: text/markdown

# wana

Audit, score and select fine-tuning datasets with evidence and file provenance.
Python 3.10+ · Python and Node packages in one repository · MIT library.

“Wana” means “Path” in WANYAM, an extinct indigenous language of the Txapacura
family. Wana connects training datasets to evidence for their curation.

## Release status

**0.1.0 is published on PyPI. This checkout prepares 0.5.0.** It implements IFD,
selection, semantic matching, the composed pipeline and the Node port. Local
validation, Alpaca-1k scoring and a three-seed trained-model pilot are documented in
[validation](docs/validation.md) and [experiments](experiments/README.md).
The pilot did **not establish superiority over random selection** (49.3%
preference, clustered 95% interval 46.8%–51.8%). See [phase status](docs/roadmap-status.md) for remaining acceptance
criteria and external publication setup.

## Install

For this checkout:

```sh
python -m pip install .
wana --version
```

After the new release is published: `pip install wana==0.5.0`.

**Normal installation includes the scoring model weights.** Wana depends on
`llm-smollm2==0.1.2`, which bundles SmolLM2-135M-Instruct Q4_1 in a roughly 93 MB
wheel, and llama-cpp-python. Once installed, default scoring is offline. Native
runtime installation may need C/C++ build tools. There are no first-use model
downloads or post-install scripts in Wana.

The model is small and predominantly English-oriented. Quantization and model
choice affect IFD ranking; IFD is not an accuracy or correctness score. The
[model decision](docs/adr/002-model-and-monorepo.md) compares packaging options,
limitations and licenses. This user-requested default supersedes the original
plan's zero-dependency installation.

Optional adapters:

```sh
pip install 'wana[score]'   # Transformers + PyTorch; choose another scoring model
pip install 'wana[embed]'   # ONNX Runtime + tokenizer for local sentence embeddings
pip install 'wana[tokens]'  # tiktoken counter for the Python API
pip install 'wana[all]'
```

## CLI

```sh
# Audit overlap. Exit 1 on EXACT or NEAR hits, 2 on invalid input/I/O.
wana check train.jsonl --eval valid.jsonl -o audit/ --markdown

# Default: installed SmolLM2 model. --resume caches losses by content and model identity.
wana score train.jsonl -o scored.jsonl --resume
wana score train.jsonl -o scored.jsonl --provider transformers_cpu \
  --model HuggingFaceTB/SmolLM2-135M-Instruct --revision MODEL_COMMIT

# --by is the ranking score; --strata is the metadata field.
wana select scored.jsonl -o selected/ --keep 0.2 --by ifd
wana select scored.jsonl -o selected/ --keep 200 --selector stratified --strata source
wana select scored.jsonl -o selected/ --keep 0.2 --diverse --threshold 0.9

# Full pipeline, with parent manifest automatically linked.
wyra build docs/*.md -o dataset --valid 0.1
wana run dataset/ -o selected/ --eval dataset/validation.jsonl --keep 0.2 --resume
wana explain selected/manifest.json
```

`--provider fake` is for tests only. `--scorer length` skips model execution;
use `--by length` when selecting those scores. `WANA_PROVIDER`, `WANA_MODEL`
and `WANA_CACHE_DIR` supply defaults. CPU context defaults to 2048 tokens; long
examples fail explicitly, and `--max-tokens` configures the limit. IFD uses the
final assistant response and the preceding messages as context. Scores above
1 are flagged and excluded by default; `--include-ifd-above-one` retains them.

`--keep 1` means one record; `--keep 1.0` means all. Fractional budgets round
down. Reapplying a fractional budget reduces the current input again; fixed-count
top-k selection is idempotent. Ties use original order, independent of seed.
Stratified selection uses largest-remainder quotas over eligible records.
Diversity accepts only examples below the cosine similarity threshold; it may
return fewer than the requested budget. Default hashing embeddings are lexical
TF-IDF vectors fitted on the input corpus, not semantic representations.

### Semantic check (opt-in)

```sh
wana check train.jsonl --eval valid.jsonl -o audit/ --semantic \
  --embedding-model /path/to/model_quantized.onnx --tokenizer /path/to/tokenizer.json \
  --semantic-threshold 0.9 --fail-on EXACT NEAR SEMANTIC
```

Supply a local ONNX sentence encoder with `input_ids` and optional
`attention_mask`/`token_type_ids`, and a Hugging Face tokenizer JSON. The adapter
supports pooled sentence output or attention-mask mean pooling of token output.
It was tested with a pinned all-MiniLM-L6-v2 ONNX snapshot. Embedding tokenization
truncates to 512 tokens. `select` also accepts `--embedder onnx` with those paths.

## Artifacts and Python API

`run` writes:

| File | Content |
| --- | --- |
| `scored.jsonl` | Original records + `wana.scores`, flags and every KEEP/DROP reason |
| `selected.jsonl` | Kept records, in original order, with annotations |
| `contamination.json` | Evidence against evaluation sets for the selected subset |
| `manifest.json` | Input/output SHA-256, parameters and optional parent |
| `report.md` | Human-readable contamination evidence |

Standalone `score` and `select` write `<output-stem>.manifest.json`; `select`
also writes `<output-stem>.decisions.json` for all input records. A sidecar
manifest beside the input takes precedence over its directory's `manifest.json`.
A failing `run` writes artifacts for inspection; it does not automatically delete
contaminated records. Outputs in the destination are replaced; input and parent
paths are protected against collisions.

```python
from wana import check_contamination, score_dataset, select_subset, run, verify_manifest

scored = score_dataset("train.jsonl")
selection = select_subset(scored, keep=0.2)
report = check_contamination([row.example for row in selection.examples], eval_sets=["valid.jsonl"])
result = run("dataset/", out_dir="selected", eval_sets=["dataset/validation.jsonl"])
assert verify_manifest("selected/manifest.json").ok
```

Public operations accept paths or loaded dataclasses. Scorers, selectors and
matchers are injected through small Protocol interfaces. Custom components can
use `Registry` factories without inheritance. An in-memory `run` returns typed
results; it does not invent file input hashes or a file manifest.

Manifest verification checks direct artifact hashes and the parent file, not
cryptographic authenticity or recursive upstream integrity. New pipeline
manifests use absolute file paths and need those files to remain accessible.
The 0.1 check format retains cwd-relative paths. Creation timestamps vary.
Original dataset metadata is retained; reports contain source text.

## Formats, determinism and limits

Automatically detects text-only OpenAI chat (`messages`), Alpaca
(`instruction`, optional `input`, `output`) and ShareGPT (`conversations`).
Python reads UTF-8 JSONL, gzip, xz and bzip2. Invalid content and unknown roles
raise contextual errors. All message contents are compared, including system
messages. `--ignore-template TEXT` removes repeated literal text after NFC/case
normalization; it is not a regex or a template parser.

EXACT means a shared normalized word n-gram (default 13), not identical whole
records. NEAR uses 64 MinHash permutations/16 LSH bands and verifies candidates
with exact shingle Jaccard (default 5-word shingles, threshold 0.8). LSH can miss
near matches. Records shorter than each matcher size produce no hits at that
level. A pair may carry evidence from multiple levels. Empty datasets are valid.
`report.ok` means no detector found a configured violation, not proven absence
of leakage. The default failure policy excludes SEMANTIC unless requested.

Readers stream, but scoring/selection and matching materialize their datasets,
indexes and results. Memory grows with inputs and matches. Pure reports are
stable for the same path strings, order, parameters and supported Unicode
normalization; model floating-point losses can vary across platforms.

## Node package

`node/` contains **@sauloleite/wana**. It implements EXACT/NEAR checks and manifest
verification, with shared Python/Node fixtures. It supports plain JSONL/gzip and
does not include IFD, embeddings or a model. See [Node README](node/README.md).
Python and npm publication are independent; npm's first publication requires
account/scope access and publisher configuration.

## Development

```sh
pip install -e '.[dev]'
ruff check .
ruff format --check .
mypy wana
pytest --cov --cov-report=term-missing
python -m build
python -m twine check dist/wana-0.5.0*
cd node && npm install && npm test
```

CI covers Python 3.10–3.13 on Linux/macOS/Windows, shared Node goldens, real
Wyra integration and an optional-runtime Linux job. PyPI publication runs on
pushes to `main` after those jobs pass. [Publication setup](docs/publishing.md).
