framework-agnostic ML diagnostics

See through your model.

ml-xray x-rays the things that actually move model quality — leakage, drift, label noise before training, and where a model fails and how its embeddings shifted after. Arrays and DataFrames in, findings out. Never a model object.

$ pip install ml-xray
See what it finds ↓ View on GitHub
6DATASET CHECKS
3DIAGNOSTIC MODULES
0MODEL OBJECTS REQUIRED
1SELF-CONTAINED HTML REPORT
the whole lifecycle

One engine, from raw data to a shipped model.

Most tools watch one moment. ml-xray spans the arc — catching the data problems that poison training, then the failure modes that only appear after.

Pre-training · dataset QA

Lint the data before it poisons the model

Leakage, distribution drift, mislabeled rows, duplicates, imbalance, and outliers — each a structured finding with a severity, the offending column, and the exact rows.

ml_xray.lint
Post-training · error analysis

Find where the model quietly fails

Automatic slice discovery surfaces the sub-populations where accuracy collapses — ranked, significance-tested, and corrected for multiple comparisons.

ml_xray.slices
Post-training · representation drift

See what moved between two embedding spaces

Compare v1 vs v2 (or embeddings over time): neighbor overlap for local structure, per-point drift for which items moved, and Adjusted Rand Index for global structure.

ml_xray.embed

// what ml-xray is NOT

  • Not a training framework. It analyzes data and predictions you already have — it never trains your model.
  • Not experiment tracking. MLflow / Weights & Biases / Aim own that.
  • Not schema matching. Valentine owns that.
  • Not a generic profiler. ydata-profiling owns exhaustive column stats; ml-xray is opinionated toward decisions that change a model.
module 01 · pre-training

Dataset linting

Point it at a DataFrame and a target. Every check returns findings with a severity you can gate CI on — bool(report) is False the moment an ERROR exists.

6 CHECKS

Catch the silent data killers

Opinionated toward the problems that actually degrade a trained model — not exhaustive statistics.

  • leakage — ~1.0 feature/target correlation, deterministic predictors, and train/test row overlap
  • drift — per-column PSI, KS test, and Jensen–Shannon divergence across splits
  • label_noise — out-of-fold confidently-wrong rows (cleanlab when installed)
  • duplicates — exact rows and MinHash/LSH near-duplicate clusters
  • imbalance — class ratio, rare levels, single-value columns
  • outliers — robust-z / IQR, high-null and constant columns
lint report · train_test.csvgate: FAIL
ERROR leakage 80 rows appear in more than one split — train/test overlap leaks labels.
WARN duplicates 40 exact duplicate rows beyond first occurrence.
WARN label_noise 17 rows (2.7%) look mislabeled — the model confidently disagrees.
INFO duplicates 40 near-duplicate clusters (80 rows, Jaccard ≥ 0.8). 
# drop-in data gate for any training pipeline
$ ml-xray lint data.csv --target y --split split --fail-on error
# exits non-zero → CI stops before you train on leaked data
module 02 · post-training

Slice discovery

A model at 84% overall can be at 47% on a sub-population you never checked. ml-xray finds those slices automatically and tells you which are real.

LATTICE SEARCH + FDR

Where the average hides the failure

Discretize features, enumerate conjunctions with Apriori-style pruning, score each slice, and keep only what survives a Benjamini–Hochberg correction across every slice tested.

  • Statistically honest — every slice carries support size, effect size, and a corrected p-value
  • Ranked by impact — |underperformance| × log(support), worst first
  • No redundant slices — children explained by a parent are pruned away
slice report · accuracybaseline 0.841
baseline 0.84 overall 0.84 region=US 0.82 region=EU −0.37 0.47 EU · tenure<9mo −0.39 0.45 
SliceFinder(metric="auto", max_depth=2).fit(X, y_true, y_pred)
# region=EU  n=898  acc=0.47  Δ−0.37  p=2e-291
module 03 · post-training

Embedding diff

Shipped a new embedding model? Compare it to the old one, row-aligned by id, and see exactly what moved — before it silently breaks your retrieval or recommendations.

LOCAL + GLOBAL STRUCTURE

Quantify what the update changed

Neighbor overlap measures whether local structure survived; per-point drift flags the items that moved; cluster stability (ARI) tracks the global picture. Dimension-free — the two spaces need not even share a width.

  • neighbor_overlap — mean k-NN Jaccard, 1.0 = local structure preserved
  • movers — the ids whose neighborhoods changed most
  • cluster_stability — Adjusted Rand Index between clusterings of A and B
embed-diff · k=10k-NN Jaccard overlap
1.00 overlap identical v1=v2 0.76 overlap 40 points moved 0.01 overlap unrelated spaces 
EmbeddingDiff(k=10).fit(emb_a, emb_b, ids=ids).report()
# overlap=0.76  ARI=0.88  movers=[id_41, id_9, ...]
how it's built

Design principles

01

Framework-agnostic

Arrays and DataFrames in — sklearn, XGBoost, LightGBM, PyTorch, or a CSV of predictions. Never a model object.

02

Deterministic

Everything is seeded. The same input produces the same report, every run.

03

Report-first

Every module returns a structured result object and emits a self-contained HTML section — no external asset requests.

04

Statistically honest

Findings carry support size and an effect estimate; slice discovery corrects for multiple comparisons.

05

Lazy optional deps

umap-learn, cleanlab, riskplot/plotly live behind extras and degrade gracefully when absent.

06

No network

No calls at import, analysis, or render time. What runs on your data stays on your machine.

get started

Install what you need

The core is four dependencies. Everything heavier lives behind an extra.

pip install ml-xray core
pip install "ml-xray[embeddings]" umap projections
pip install "ml-xray[noise]" cleanlab label noise
pip install "ml-xray[viz]" matplotlib / plotly
pip install "ml-xray[all]" everything
python -m pytest 59 tests, all green