Metadata-Version: 2.4
Name: nirs4all-tools
Version: 0.0.7
Summary: Offline, one-way, no-in-place migration tools for legacy nirs4all artifacts.
Author-email: Gregory Beurier <beurier@cirad.fr>
Maintainer-email: Gregory Beurier <gregory.beurier@cirad.fr>
License-Expression: CECILL-2.1 OR AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/GBeurier/nirs4all-tools
Project-URL: Repository, https://github.com/GBeurier/nirs4all-tools
Project-URL: Issues, https://github.com/GBeurier/nirs4all-tools/issues
Keywords: nirs4all,migration,workspace,legacy,converter
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSES/AGPL-3.0-or-later.txt
License-File: LICENSES/CeCILL-2.1.txt
License-File: LICENSES/COMMERCIAL-LICENSE.md
License-File: LICENSES/COMMERCIAL-LICENSE_FR.md
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.0.0; extra == "duckdb"
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0.0; extra == "parquet"
Provides-Extra: trusted-joblib
Requires-Dist: joblib>=1.2; extra == "trusted-joblib"
Requires-Dist: scikit-learn>=1.0; extra == "trusted-joblib"
Provides-Extra: n4mm-export
Requires-Dist: pls4all<2,>=1.0.13; extra == "n4mm-export"
Provides-Extra: target
Requires-Dist: nirs4all>=0.10.0; extra == "target"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.7.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: duckdb>=1.0.0; extra == "dev"
Requires-Dist: joblib>=1.2; extra == "dev"
Requires-Dist: scikit-learn>=1.0; extra == "dev"
Dynamic: license-file

# nirs4all-tools

Offline, **one-way**, **no-in-place** migration tools for legacy `nirs4all`
artifacts (workspaces, `.n4a` bundles, loose prediction files).

This is the standalone home for the legacy *readers* that used to live inside
the `nirs4all` runtime. The V1 runtime carries **no** legacy reader and **no**
auto-migration trigger; instead, `nirs4all-tools` converts old stores **into**
the format the runtime already reads (`nirs4all-workspace-v2`), so users keep
their predictions/pipelines without the runtime ever opening a legacy store.

> Status: **first transform** (lane `L18`, lock `LOCK-MIG`, decision `DEC-MIG-001`).
> Candidate packages are **unpublished**; install a wheel supplied through the
> release process rather than assuming a package-registry version exists.
> The CLI surface, the no-in-place safety machinery, detection, the contract
> vocabulary, `inspect`, `migrate --dry-run`, and `--copy-only` are implemented.
> The first schema transform lowers `sqlite-workspace-legacy-arrays` metadata
> into a fresh workspace-v2 `store.sqlite`; legacy array rows are lowered into
> runtime-readable `arrays/<dataset>.parquet` sidecars when the optional
> `parquet` extra is installed, and the raw rows are still preserved as
> checksummed JSONL audit provenance. A native-results-v1 preview can lower one
> current dag-ml native results directory into runtime-readable workspace-v2
> metadata plus array sidecars after strict hash/schema preflight. One closed
> historical DuckDB profile can lower a validated multi-run/pipeline/chain graph
> with flat arrays after read-only schema and relation preflight. A legacy
> `runs/*/*/manifest.yaml` preview can lower one completed run when it references
> one complete `*_predictions.json` payload and the YAML/JSON metadata agree.

## The one contract: no-in-place

Every command guarantees the source is never modified:

- the source is opened **read-only** (SQLite via `file:…?mode=ro&immutable=1`);
- `--output` is **mandatory** and must be **disjoint** from the input
  (aliasing / nesting is refused, exit `40`);
- the output must be **empty**; `--resume` is only a read-only, attested no-op
  for an already complete output, never a continuation of a partial migration;
- `inspect` and `migrate` materialize a private descriptor-bound, no-follow
  source view before detection, parsing, transforms, or payload copies; the
  original source is rechecked before completion (a mismatch is exit `70`);
- that secure materialization requires POSIX `O_NOFOLLOW`, `O_DIRECTORY`, and
  descriptor-relative (`dir_fd`) operations.  Where they are unavailable,
  source-consuming commands fail closed with unsupported capability rather
  than fall back to path traversal.  It needs temporary space roughly equal
  to the source, and `TMPDIR` must resolve outside both source and output.

## Install

```bash
pip install -e ".[dev]"          # scaffold core is pure standard library
pip install -e ".[duckdb]"       # add DuckDB-source reading (optional)
pip install -e ".[parquet]"      # add Parquet lowering/validation (optional)
pip install -e ".[trusted-joblib]" # explicit, trusted sklearn PLS inspection only
pip install -e ".[trusted-joblib,n4mm-export]" # opt-in trusted PLS -> PREDICT-only N4MM export
```

## CLI

For an operator procedure that needs no source checkout, including the R2 to R1
rollback path, see [Workspace conversion runbook](docs/workspace-conversion-runbook.md).
The version-based read/write/migrate commitment and exact per-input dispositions
are defined by the [legacy support SLA](docs/legacy-support-sla.md) and its
[machine-readable matrix](docs/contracts/legacy-support-matrix.v1.json).

```bash
nirs4all-tools --version

# Read-only: detect what a legacy location contains.
nirs4all-tools workspace inspect INPUT [--format json|text] [--report PATH]

# Convert into a fresh workspace-v2 output (one-way, no-in-place).
nirs4all-tools workspace convert INPUT --output OUTPUT [--dry-run | --verify]

# Historical/advanced interface, retained unchanged.
nirs4all-tools legacy inspect <input> [--format json|text] [--report PATH]
nirs4all-tools legacy migrate <input> --output DIR --target nirs4all-workspace-v2 \
    [--manifest PATH] [--report PATH] [--id-map PATH] [--unsupported-report PATH] \
    [--checksums sha256] [--dry-run | --verify] [--strict | --best-effort] \
    [--copy-only] [--resume] [--trusted-load-joblib]

# Verify an output against its manifest (reads no source).
nirs4all-tools legacy verify <output-dir> --manifest PATH [--report PATH]

# Deliberately narrow: produces model.n4mm plus an attestation, never a
# workspace or archive.  joblib is deserialized only with this explicit flag.
nirs4all-tools legacy export-n4mm <trusted-pls.joblib> --output DIR --trusted-load-joblib
```

`workspace inspect` is read-only. `workspace convert` fixes the target to
`nirs4all-workspace-v2` and returns the same stable domain codes as the
historical converter: `0` for a clean conversion, `10` when unsupported items
are preserved opaque in best-effort mode, and `20` when unsupported input is
refused in strict mode. Both commands leave the source path, inode, and bytes
untouched; they never rename it or create a `.bak` copy.

Current schema-transform support is intentionally narrow:

`--resume` is deliberately not crash recovery. It accepts only an output whose
four default in-output contracts are complete and mutually consistent, whose
source path/fingerprint, target, and migration mode match the current command,
and whose checksums and output inventory still verify. It does not write any
file and returns the prior terminal code (`0` or `10`). External contract paths
are never resume authority. Missing, external, malformed, partial, or
mismatching contract state is refused with code `20`.
Here, *attested* means internally consistent contracts, inventories, and
checksums; it is not a cryptographic signature or protection against an actor
that can rewrite every output contract and payload coherently.

For a real migration, the only contract paths allowed inside `--output` are
the four default root files. Custom manifest, report, id-map, or unsupported
report paths must be external to `--output`; this keeps later standalone
`legacy verify` checks unambiguous.

`--copy-only` preserves bytes only when source entry names are representable in
the portable slash-separated checksum ledger. In particular, a POSIX filename
containing a literal backslash is refused before output creation rather than
creating an output that cannot later verify.

Before `--copy-only` creates its private source stage, it reserves the logical
size of every source file (including sparse files and each hard-link name) on
the relevant `TMPDIR`, output, and external-contract volumes. Requests sharing
a filesystem are summed because the private source and sibling publication
trees coexist. A capacity shortfall or runtime `ENOSPC`/quota failure is a
policy refusal (exit `40`), not an invalid archive. The copied payload and all
default in-output contracts are written to a private sibling directory and
published with one directory rename; before that commit, a missing output
remains missing and an existing empty output remains empty if the run fails.
Custom external contracts are prepared and atomically replaced independently
after that output commit, because separate filesystems cannot share one global
atomic transaction.

- `legacy export-n4mm` can, only after explicit `--trusted-load-joblib`, prove
  a finite affine equation from exactly a fitted sklearn `PLSRegression` and
  export it through the public `pls4all` binding (Methods ABI 2.3) as a native
  PREDICT-only `model.n4mm` plus an attestation. It refuses pipelines and
  arbitrary estimators, never runs
  automatically, and never fabricates a workspace or archive: a standalone
  joblib has no signed graph, score, cohort, or lineage evidence;

- `sqlite-workspace-legacy-arrays` metadata is lowered to `store.sqlite`
  schema v2;
- the legacy `prediction_arrays` table is decoded offline, lowered to the
  runtime array sidecar schema (`arrays/<dataset>.parquet`), and also preserved
  in `preserved/legacy-prediction-arrays.jsonl` for audit;
- one standalone `store.duckdb` in the exact historical six-table profile is
  lowered to workspace-v2 metadata plus flat runtime array sidecars when both
  the `duckdb` and `parquet` extras are installed. It accepts a closed graph of
  runs, pipelines, chains, predictions, logs, and complete finite equal-length
  arrays; it refuses artifact references, sidecar filename collisions, unknown
  source files, and broken graph ownership before output creation. The original
  database remains checksummed under `preserved/duckdb-workspace/`;
- one standalone current dag-ml `native-results-v1` directory with exact integer
  `schema_version: 3`, a valid `score_set_hash`, and canonical
  `predictions.parquet` projection is lowered to
  workspace-v2 run/pipeline/chain/prediction/artifact metadata plus
  runtime-readable `arrays/<dataset>.parquet` sidecars; the original native
  payload is still checksummed under `preserved/native-results-v1/`;
- malformed, older, mixed, or multi-artifact `native-results-v1` sources fail
  `--strict` with a machine-checkable schema/preflight cause, and best-effort
  mode preserves them opaque with the same reason in the manifest;
- one standalone complete `*_predictions.json` loose-prediction payload is
  lowered to workspace-v2 run/pipeline/chain/prediction metadata plus
  runtime-readable `arrays/<dataset>.parquet` sidecars when the `parquet` extra
  is installed; the original loose JSON and sibling metadata files are still
  checksummed under `preserved/loose-predictions/`;
- one standalone legacy `runs/*/*/manifest.yaml` tree is lowered when its single
  manifest points to one complete `*_predictions.json` under the same source
  root and `run_id`, `pipeline_id`, dataset, model, and preprocessing metadata
  match; the manifest tree and referenced prediction payload remain
  checksummed under `preserved/`;
- a `.n4a` is preserved only after a bounded structural ZIP preflight: one
  finite JSON-object `manifest.json`, portable relative member names, no
  encrypted/symlink/special members, no normalization collisions, and explicit
  archive/member/expansion limits. It is copied opaque under `preserved/` and
  is never extracted, deserialized, or made predictive; `.n4a.py` and
  non-lowerable `native-results-v1` artifacts remain opaque checksummed payloads
  with an empty workspace-v2 store;
- non-lowerable DuckDB variants, legacy `runs/` trees outside the
  single-manifest preview, incomplete or mixed loose prediction files, and
  already-v2 SQLite stores are preserved opaque by default in best-effort
  mode; `--strict` refuses them before writing;
- every real migration writes `unsupported-report.json` alongside the manifest,
  report, and id-map; dry runs write the same machine-readable unsupported
  report only when `--unsupported-report PATH` is provided;
- best-effort migration exits `10` only when semantic lowering is unavailable
  and content must be preserved opaque;
- `--strict` requires semantic lowering and exits `0` for fully lowered array
  sources or native-results metadata previews.

### Exit codes

| Code | Meaning |
|------|---------|
| `0`  | success, no warnings |
| `10` | migrated with warnings (best-effort preserved opaque / non-fatal skips) |
| `20` | unsupported input (unknown, unsafe archive, forward-version source, strict unsupported item, or unattested `--resume`) |
| `30` | verification failed |
| `40` | refused by policy (in-place / aliased output, non-empty fresh output, or insufficient storage) |
| `70` | internal error (incl. source-tree integrity assertion failure) |

## Contracts

Four durable JSON contracts are emitted alongside a migrated workspace
(`SW4_MIG_CONVERTER_spec.md` §7–10):

- `legacy_migration_manifest.v1` — the exhaustive inventory + checksum + id-map ledger;
- `legacy_migration_report.v1` — the human/UX digest + next action;
- `legacy_id_map.v1` — the never-lossy old→new id map.
- `legacy_unsupported_report.v1` — the machine-readable list of unsupported,
  refused, or opaque-preserved items.

## Development

```bash
ruff check .
mypy
pytest
```

Checked-in converter goldens live under `tests/fixtures/legacy/`. They are
small reduced legacy payloads for old workspaces, run/pipeline manifests, and
prediction arrays. Tests copy or materialize them into temporary directories
before migration so the source goldens stay read-only and the no-in-place
contract remains observable.

## License

Dual-licensed **CeCILL-2.1 OR AGPL-3.0-or-later** (plus commercial), consistent
with the nirs4all ecosystem policy. See `LICENSE` for the summary and
`LICENSES/CeCILL-2.1.txt` / `LICENSES/AGPL-3.0-or-later.txt` for the complete
canonical texts. Contact:
`nirs4all-admin@cirad.fr`.
