Metadata-Version: 2.5
Name: zekan
Version: 0.1.1
Summary: A local-first ML trust-audit tool for finding and measuring data leakage
Project-URL: Homepage, https://github.com/91Sakthivel/Zekan-Audit
Project-URL: Repository, https://github.com/91Sakthivel/Zekan-Audit
Author: 91Sakthivel
License: MIT
License-File: LICENSE
Keywords: data-leakage,data-science,machine-learning,ml-testing,model-validation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: <3.14,>=3.10
Requires-Dist: jinja2>=3.1
Requires-Dist: joblib>=1.3
Requires-Dist: numpy<3.0,>=1.24
Requires-Dist: pandas<4.0,>=2.0
Requires-Dist: psutil>=5.9
Requires-Dist: pyarrow>=14.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: scikit-learn<1.10,>=1.9
Requires-Dist: threadpoolctl>=3.5.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# Zekan

**Find out if your model is secretly cheating — and how much it's cheating.**

When you build a machine-learning model, information can accidentally leak into it that it wouldn't have in the real world. The model looks great in testing, then falls apart in production. This is called **data leakage**, and it's one of the most common and expensive mistakes in machine learning.

Most tools just tell you *whether* there's a leak. Zekan tells you what actually matters:

- **How much** the leak is costing you (measured as a drop in your model's real score)
- **Which feature** is causing it
- **Whether it's real** or just statistical noise
- **What Zekan did *not* check** — so a clean result never gives you false confidence

Zekan runs entirely on your own machine. Your data never leaves your computer.

Validated on two independent real-world datasets from different domains — a
hospital-readmission benchmark and a 91,323-row mortgage-servicing panel — with
every prediction stated before the second run, not fitted after the fact. See the
[validation study](zekan/benchmark/results/DATASET2_FREDDIEMAC_VALIDATION_STUDY.md).

---

## Does Zekan fit my project?

Zekan works when **all** of these are true:

- You're predicting a **yes/no outcome** (e.g. will this customer churn? will this patient be readmitted?).
- Your data is a **table** (rows and columns — CSV or Parquet).
- The same thing appears **more than once over time** — e.g. one customer with several monthly snapshots. Zekan uses this time structure to detect leaks.

If your data has one row per item with no repeats over time, Zekan's core check won't apply yet.

---

## The idea in one line

Zekan trains your model the normal way, then trains an **honest** version with the leak removed, and reports the difference. That difference is the performance that was fake — the part that would vanish in production.

---

## Install

You'll need Python 3.10 or newer.

```bash
# Get the code
git clone https://github.com/91Sakthivel/Zekan-Audit.git
cd Zekan-Audit

# Set up an isolated environment (recommended)
python -m venv .venv

# Turn it on:
#   Windows (PowerShell):
.venv\Scripts\Activate.ps1
#   macOS / Linux:
source .venv/bin/activate

# Install Zekan
pip install -e ".[dev]"
```

Once installed, you can type `zekan` as a command.

---

## Try it in 60 seconds (no setup)

A ready-made example ships with Zekan, so you can see it work before using your own data. This runs `--tier scan` — structural checks only, no model fit — so it's fast regardless of your machine:

```bash
zekan audit --tier scan --data examples/churn_instacart/fixture.csv --config examples/churn_instacart/zekan.yml
```

You'll get a plain-language structural report, not a leakage verdict yet — a scan never fits a model, so it can't measure `fl` or confirm anything is clean. For the real verdict, run the full audit (Step 2 below).

---

## A small worked example

Let's walk through that example so the pieces make sense. The data (`fixture.csv`) is a tiny customer-churn table — a few columns, one row per customer per month:

| customer_id | snapshot_date | tenure_months | spend_last_30d | leaky_col | churned |
|---|---|---|---|---|---|
| 1001 | 2023-01-31 | 5 | 42.0 | ... | 0 |
| 1001 | 2023-02-28 | 6 | 38.5 | ... | 1 |
| 1002 | 2023-01-31 | 22 | 110.0 | ... | 0 |

Here's how each column maps to what Zekan asks you:

- **`customer_id`** → the **entity_id**. It's the same customer appearing across several months — that's the "recurring over time" structure Zekan needs.
- **`snapshot_date`** → the **prediction_time**. Each row is a monthly snapshot.
- **`churned`** → the **target**. The yes/no thing we're predicting (1 = churned, 0 = stayed).
- **`spend_last_30d`, `tenure_months`** → normal features. Fair game — they'd be known at prediction time.
- **`leaky_col`** → declared **forbidden**. In this example it's a stand-in for a column that gives away the answer, so we tell Zekan it must not be used.

The matching config (`zekan.yml`) simply writes those choices down:

```yaml
contract:
  entity_id: customer_id
  prediction_time: snapshot_date
  target: churned
  available_features_until: snapshot_date
  forbidden_after_prediction:
    - leaky_col
```

When you run the audit, Zekan checks whether that forbidden column (or any other) is secretly leaking, measures how much, and prints a verdict — for this clean example, **TRUSTED**. Swap in your own data and your own column names, and it's the same three steps below.

---

## Use it on your own data (3 steps)

### Step 1 — Let Zekan set up a config for you

```bash
zekan init --data your_data.csv
```

Zekan reads your file, lists your columns, and asks you a few questions. For each one you can type **the column name or its number** — whichever is easier. It asks for:

- **entity_id** — the column that identifies *one thing* tracked over time (e.g. `customer_id`).
- **prediction_time** — the column with the date/time of each row (e.g. `snapshot_date`).
- **target** — the yes/no thing you're predicting (e.g. `churned`).
- **available_features_until** — usually the same as your time column. It means "only information known up to this point is fair to use."
- **forbidden_after_prediction** — any columns that would *give away the answer* if the model used them. Leave empty if you're not sure; you can add them later. (You can list several, by name or number.)

This writes a file called `zekan.yml`.

> **Not sure what counts as "forbidden"?** A forbidden feature is anything that couldn't actually be known at the moment you'd make the prediction — for example, a field that's only filled in *after* the outcome happens. If in doubt, leave it empty and run the audit; Zekan also screens for leaks you didn't declare.

### Step 2 — Run the audit

```bash
zekan audit --data your_data.csv --config zekan.yml
```

That's it — no other editing needed. Zekan runs the full check and prints a verdict.

### Step 3 — Read the verdict

Zekan gives you one of four results:

| Verdict | What it means |
|---|---|
| **TRUSTED** | No leakage found *in what you declared and what Zekan checks*. (Not a guarantee the whole pipeline is perfect — Zekan tells you its scope.) |
| **RISKY** / **FAILED** | A real, confirmed leak — with how much it's costing you and which feature is responsible. |
| **INCONCLUSIVE** | The result wasn't stable enough to trust. |

---

## What the verdict is measuring

- **`fixable_leakage` (`fl`) = AUC(B) − AUC(C)** — model B keeps every
  column you declared forbidden; model C has them removed. Both are
  scored on the identical, honest, forward-in-time fold split, so the
  only thing that differs is whether the model got to see those columns.
  The gap is the accuracy that would evaporate in production, and Zekan
  attributes it back to whichever feature(s) caused it.
- **Two different questions, not one number.** `p` / `NSL` (from the
  permutation null) answer *"is this leak distinguishable from noise?"* —
  that's statistical power, not damage. `fl` answers *"how much is it
  costing you?"* — that's severity. A confidently-real leak can still be
  small; a large leak can still be statistically unconfirmed at small
  sample sizes. Don't read NSL as a severity score — it isn't one.
- **`evidence_level`: `SCREENED` / `AUDITED` / `CERTIFIED`** — how much of
  the check actually ran, separately from the verdict itself. `--tier
  scan` → `SCREENED` (structural checks only, no `fl`). `--tier audit` →
  `AUDITED` (`fl` is computed, but no permutation null ran, so nothing
  was confirmed real). `--tier certify` → `CERTIFIED` (the full check,
  including the null). Because `AUDITED` never ran the null, it can never
  resolve to a bare `TRUSTED` — a low `fl` at that tier reports
  `UNCONFIRMED_LOW_DAMAGE` instead, so a clean-looking audit-tier result
  can't be mistaken for a confirmed-clean one.
- **`--with-precision`** (audit tier only) turns on the bootstrap
  confidence interval around `fl` at `--tier audit`; it's off there by
  default for speed. `--tier certify` always computes it.

See [`METHODOLOGY.md`](METHODOLOGY.md) §1–3 for the full A/B/C
decomposition and the exact formulas.

---

## Getting a machine-readable result (for automation)

To use Zekan in a CI pipeline or script, add `--json`:

```bash
zekan audit --data your_data.csv --config zekan.yml --json
```

This prints structured JSON (the human-readable text goes to the error stream instead), so another tool can read the verdict automatically.

---

## Commands at a glance

| Command | What it does |
|---|---|
| `zekan init` | Ask a few questions and write a config for you. |
| `zekan audit` | Run the leakage audit and print a verdict. |
| `zekan diff` | Compare two audits to see if leakage got better or worse. |
| `zekan report` | Produce a report from an audit. |
| `zekan explain <feature>` | Show everything Zekan already knows about one column, from a saved audit JSON (`--json`) or a fresh run (`--config`). No new measurement. |
| `zekan benchmark` | Run Zekan's built-in test suite. |

Useful `zekan audit` options (run `zekan audit --help` for all of them):

| Option | What it does |
|---|---|
| `--json` | Machine-readable output for automation. |
| `--stability` | Rerun the permutation null across N seeds; downgrade to INCONCLUSIVE if the verdict depends on which null draws were sampled. Does not reseed model fitting -- see "Honest limitations" below. |
| `--dry-run` | Check your config is valid without running the full audit. |
| `--estimator NAME` | Choose the model type (`histgb` by default). |
| `--sep CHAR` | CSV delimiter, e.g. `--sep '|'`. Zekan also auto-detects a non-comma delimiter and tells you what it found, but `--sep` is the reliable override. |
| `--from-period VALUE` | Restrict the audit to rows at or after this value in your `prediction_time` column — useful for re-running on recent data without rebuilding a whole frame. |

---

## Performance: what each tier costs, and why

`zekan audit` (the default, `--tier certify`) fits roughly **200 models**
per run: a real A/B/C leakage decomposition, plus a permutation null (2
channels × 100 draws) that confirms the leak is statistically real, not a
one-off fold split. That cost is intrinsic to what certify measures, not
an oversight — eight separate attempts to cut it were tried and refuted
against real data (a cheaper model changed a verdict; subsampling swung
the detection margin; several others failed their own correctness checks
before ever reaching a speed measurement). See
[`METHODOLOGY.md`](METHODOLOGY.md) §6 for the full cost model, every
figure's actual measurement scale (91,323 and 944,355 rows are real
measurements; anything larger is a labeled projection, not a
measurement), and why `--jobs` (and therefore wall-clock) is
hardware-dependent by design.

Lighter tiers exist for faster iteration: `--tier scan` (structural checks
only, no model fits) and `--tier audit` (the A/B/C decomposition, no
permutation null — no statistical confirmation of detection either).

**Results, not just cost, were checked across hardware.** The same
944,355-row run produced byte-identical `fl` and bootstrap CI values on a
laptop and on a GCP Sapphire Rapids instance, despite NumPy/SciPy
dispatching a different OpenBLAS kernel on each. One CPU pair, not a
universal guarantee — see
[`zekan/benchmark/results/CLOUD_CROSS_HARDWARE_REPRODUCIBILITY_FINDINGS.md`](zekan/benchmark/results/CLOUD_CROSS_HARDWARE_REPRODUCIBILITY_FINDINGS.md)
for the full record.

**The tested stack**, exactly, across both 944,355-row runs above:
Python 3.12 (3.12.3–3.12.4), NumPy 2.4.6–2.5.2, SciPy 1.18.0–1.18.1, and
scikit-learn **1.9.0 exact** on both machines — scikit-learn is the
version-sensitive one (see `pyproject.toml`'s own `<1.10` upper bound),
so this is the range actually exercised, not merely permitted.

---

## Honest limitations

Zekan is under active development, and we'd rather tell you its limits than oversell it:

- It's been validated on real data (a large public hospital-readmission dataset, plus a second, independent mortgage-servicing dataset) through experiments designed in advance — but broader testing across more datasets is still in progress.
- **The severity thresholds (`warn_floor`/`fail_floor`) were calibrated on one dataset, and a second-dataset test found they do NOT transfer as absolute numbers.** The same injected leak that Zekan correctly detected and correctly ranked as more severe than a clean control on both datasets only crossed `warn_floor` on the dataset the thresholds were calibrated against — on the second dataset, a real, confirmed leak topped out at roughly a sixth of `warn_floor`, because that dataset's honest model ceiling left far less headroom for any leak to register in. Detection and relative ordering (clean vs. leaky) held on both; the absolute floor did not. See the [validation study](zekan/benchmark/results/DATASET2_FREDDIEMAC_VALIDATION_STUDY.md) §2 and [`METHODOLOGY.md`](METHODOLOGY.md) §4 for the full account.
- It needs data where the same entity recurs over time; it doesn't fit ordinary one-row-per-item datasets yet.
- Version 1 focuses on yes/no predictions on tabular data.
- **`--stability` only checks whether the verdict depends on which permutation-null draws were sampled.** It does not measure model-fitting variance, n-sensitivity, or cross-dataset drift. A dedicated model-refit-variance check was designed and measured, then deliberately not shipped: for the default estimator at the scale of every real dataset this project has tested, refitting under a different random seed changed nothing at all — not "stable," but not a check that has anything to detect at that scale either. See [`STABILITY_CI_PREREGISTRATION.md`](zekan/benchmark/results/STABILITY_CI_PREREGISTRATION.md).

Every verdict Zekan gives includes a note about what it did and did not check. That honesty is the point — see [`COVERAGE_MAP.md`](COVERAGE_MAP.md) for the full, itemized account of what Zekan checks, what it doesn't, and the known boundaries of each.

---

## License

MIT — see [LICENSE](LICENSE). Free to use, modify, and share.
