Metadata-Version: 2.5
Name: ml-leakproof
Version: 0.2.0rc1
Summary: Leakproof: static, dataset, and runtime leakage/evaluation-rigor checks for ML workflows.
Project-URL: Homepage, https://github.com/Arianhgh/Leakproof
Project-URL: Documentation, https://github.com/Arianhgh/Leakproof#readme
Project-URL: Issues, https://github.com/Arianhgh/Leakproof/issues
Author: Arianhgh
License: MIT
License-File: LICENSE
Keywords: ci,data-leakage,linter,machine-learning,ml-evaluation,static-analysis
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: nbformat>=5.0
Requires-Dist: rich>=13.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: typer>=0.9
Provides-Extra: cst
Requires-Dist: libcst>=1.0; extra == 'cst'
Provides-Extra: data
Requires-Dist: datasketch>=1.5; extra == 'data'
Requires-Dist: imagehash>=4.3; extra == 'data'
Requires-Dist: numpy>=1.23; extra == 'data'
Requires-Dist: pandas>=2.0; extra == 'data'
Requires-Dist: pillow>=9.0; extra == 'data'
Requires-Dist: pyarrow>=12.0; extra == 'data'
Requires-Dist: scikit-learn>=1.1; extra == 'data'
Requires-Dist: scipy>=1.9; extra == 'data'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: hatch>=1.14; extra == 'dev'
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: jsonschema>=4.20; extra == 'dev'
Requires-Dist: libcst>=1.0; extra == 'dev'
Requires-Dist: mypy==2.3.1; extra == 'dev'
Requires-Dist: numpy>=1.23; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff==0.16.7; extra == 'dev'
Requires-Dist: scikit-learn>=1.1; extra == 'dev'
Requires-Dist: scipy>=1.9; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Requires-Dist: wrapt>=1.14; extra == 'dev'
Provides-Extra: fix
Requires-Dist: libcst>=1.0; extra == 'fix'
Provides-Extra: llm
Requires-Dist: anthropic>=0.25; extra == 'llm'
Requires-Dist: openai>=1.0; extra == 'llm'
Provides-Extra: runtime
Requires-Dist: numpy>=1.23; extra == 'runtime'
Requires-Dist: scikit-learn>=1.1; extra == 'runtime'
Requires-Dist: wrapt>=1.14; extra == 'runtime'
Provides-Extra: runtime-integrations
Requires-Dist: imbalanced-learn>=0.10; extra == 'runtime-integrations'
Requires-Dist: lightgbm>=3.3; extra == 'runtime-integrations'
Requires-Dist: xgboost>=1.7; extra == 'runtime-integrations'
Description-Content-Type: text/markdown

# Leakproof

Leakproof is a static, dataset, and runtime checker for data leakage and
evaluation-rigor mistakes in machine-learning workflows. The release candidate
is `0.2.0rc1`; the distribution is `ml-leakproof`, the Python package is
`ml_leakproof`, and the executable is `ml-leakproof`.

## Install

This candidate is **unpublished**. Install from this checkout (Python 3.10–3.14):

```bash
git clone https://github.com/Arianhgh/Leakproof.git
cd Leakproof
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
python -m pip install .                         # static analysis
python -m pip install ".[data,runtime,fix]"      # dataset, runtime, and autofix features
python -m pip install ".[llm]"                   # optional explanation providers
```

The `data` extra includes CSV and Parquet support. Additional runtime framework
hooks use `.[runtime,runtime-integrations]`. For an unpublished wheel, use
`python -m pip install "./dist/ml_leakproof-0.2.0rc1-py3-none-any.whl[data,runtime,fix]"`.
The similarly named `leakproof-ml` is a different project.

The base install does not import or require pandas, scikit-learn, runtime
integrations, or LLM SDKs. `python -m ml_leakproof` is equivalent to the
`ml-leakproof` command.

## Quickstart

```bash
ml-leakproof check .
ml-leakproof check . --format sarif --output leakproof.sarif --fail-on high
ml-leakproof run train.py --format json --output runtime.json
ml-leakproof audit-data --train train.csv --test test.csv --target label
ml-leakproof rules
ml-leakproof explain P001
ml-leakproof version
```

`check` is static-only. `run` executes one script inside runtime instrumentation;
the script's own exit status is preserved in the result. Script `stdout` is
forwarded to `stderr` so machine-readable runtime reports remain valid JSON or
SARIF. `audit-data` validates
explicit pandas train/validation/test frames and never executes model code.

Runnable static, runtime, and data-layer examples are in
[`examples/README.md`](examples/README.md).

## Result contract

Result-oriented APIs return `AnalysisResult`, not an unqualified list:

```python
from ml_leakproof import AnalysisError, analyze, analyze_data, check

result = analyze("src")
print(result.findings)
print(result.diagnostics)
print(result.coverage, result.completion)  # complete | partial | failed

try:
    findings = check("src")
except AnalysisError as error:
    # The partial result is available for inspection and serialization.
    print(error.result.coverage, error.result.diagnostics)

data_result = analyze_data(train, test, val=validation, target="label")
```

The list wrappers `check` and `audit_data` raise `AnalysisError` when requested
analysis is incomplete unless `allow_partial=True` is supplied in the call or
configuration. Partial results are never silently presented as complete.

## Exit codes and profiles

| Code | Meaning |
|---|---|
| `0` | No gateable finding at or above the configured severity gate |
| `1` | At least one gateable finding meets the severity and confidence gate |
| `2` | Usage/configuration error, failed analysis, incomplete analysis, or non-zero script exit |

`--min-confidence` controls display only. It does not change the gate decision.
`--gate-confidence` controls the minimum confidence required to fail a gate.
The built-in profiles are:

| Profile | Display threshold | Gate threshold |
|---|---:|---:|
| `ci` | `0.60` | `0.75` |
| `notebook` | `0.30` | `0.75` |
| `research` | `0.00` | `0.75` |

Findings marked `advisory_only`—for example similarity and target-predictivity
probes, grouped/time hints, and reproducibility/reporting suggestions—are
visible evidence but never gate a run. A clean scan means that no checked rule
produced a gateable finding; it is not proof that a methodology is valid.

## Configuration and suppressions

Keep the dedicated `leakproof.toml` filename, or use `[tool.leakproof]` in
`pyproject.toml`. CLI flags override project configuration. Unknown keys and
invalid ranges fail loudly. LLM use is disabled by default and requires an
explicit provider and model.

```toml
[tool.leakproof]
profile = "ci"
select = ["ALL"]
fail_on = "high"
min_confidence = 0.60
gate_confidence = 0.75
layers = ["static", "data", "runtime"]
```

Suppress only a reviewed, confirmed false positive:

```python
# leakproof: ignore[P001]
```

File-wide suppression is available with `# leakproof: ignore-file`; malformed
or unknown directives are reported as diagnostics. Suppression markers inside
strings are not interpreted as directives.

## What is checked

The built-in catalog has 30 rules spanning split hygiene, cross-validation,
preprocessing, overlap, target leakage, temporal boundaries, test-set
adaptivity, metrics, and determinism. See the [rule catalog](docs/rules.md) or
run `ml-leakproof rules --format json`.

Static analysis is order- and scope-aware, understands ordinary and keyword
arguments, reassignment, aliases, pipelines, and common sklearn adapters. It
does not execute source code. Notebook cells are analyzed in stored order;
execution history and out-of-order state are not reconstructed, and malformed
or magic cells are isolated with coverage diagnostics.

The data layer checks every declared split pair, including validation, with
type-aware exact row hashes. Near-duplicate, target-predictivity, mutual
information, and imbalance checks are bounded and advisory; capped or
unavailable work is recorded in `coverage` and `diagnostics`. Input data stays
in the current process unless the explicitly enabled LLM explanation layer is
used; datasets are not sent to an LLM by the data audit itself.

Runtime tracking supports explicit registration as well as automatic hooks for
common sklearn splitters, estimators, pipelines, CV helpers, pandas/NumPy
assembly, and supported integrations. Hooks are transactional and nested
sessions are isolated. Runtime instrumentation covers the current process and
does not reconstruct child-process execution. Import split helpers inside the
`watch()` scope when relying on automatic hooks; explicit `register_split()` is
available for integrations that cannot be patched automatically.

NumPy outputs from learned preprocessing carry session-owned, operation-based
ancestry through supported copies, casts, and row selections. Equal-valued
independent arrays are not linked; unsupported propagation is reported as
partial coverage instead of being treated as clean.

Tracked arrays serialize as ordinary NumPy arrays for pickle/joblib
compatibility. Lineage remains local to the originating watch session and is
intentionally not restored when a saved array or estimator is loaded.

## Autofix and LLM explanations

`ml-leakproof check --fix-preview` shows a unified diff. `--fix` applies only
verified LibCST edits (currently deterministic seed arguments), checks the
source digest, writes atomically, and rescans. It never rewrites notebooks or
calls that use `**kwargs`.

`--explain` may add bounded, uncertainty-aware explanations to deterministic
findings. `--llm-triage` is limited to files that failed static parsing. LLM
output cannot create, delete, or replace a deterministic finding, and is never
required for a normal scan.

## CI, plugins, and corpus benchmarks

- [GitHub Action and CI usage](docs/ci.md)
- [Rule and adapter plugins](docs/plugins.md)
- [0.2 migration guide](docs/migration-0.2.md)
- [Benchmark methodology](docs/benchmark.md)
- [Local release validation](docs/releasing.md)
- Immutable-commit [corpus manifest](corpus-manifest.json)

Third-party plugins use the `ml_leakproof.rules` and
`ml_leakproof.adapters` entry-point groups and vendor-prefixed rule IDs. See
the plugin guide for the migration from the pre-0.2 import and executable
names.

## License

MIT; see [LICENSE](LICENSE).
