Metadata-Version: 2.4
Name: DataMedicine
Version: 1.0.0
Summary: Validate, diagnose, and treat tabular datasets in seconds.
Project-URL: Homepage, https://github.com/datamedicine/datamedicine
Project-URL: Documentation, https://github.com/datamedicine/datamedicine#readme
Project-URL: Repository, https://github.com/datamedicine/datamedicine
Project-URL: Issues, https://github.com/datamedicine/datamedicine/issues
Project-URL: Changelog, https://github.com/datamedicine/datamedicine/blob/main/CHANGELOG.md
Author: DataMedicine Contributors
License-Expression: MIT
License-File: LICENSE
Keywords: data-quality,machine-learning,pandas,validation
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == 'excel'
Provides-Extra: parquet
Requires-Dist: pyarrow>=14; extra == 'parquet'
Provides-Extra: pdf
Requires-Dist: weasyprint>=61; extra == 'pdf'
Provides-Extra: viz
Requires-Dist: matplotlib>=3.7; extra == 'viz'
Description-Content-Type: text/markdown

# DataMedicine

> Validate, diagnose, and treat tabular datasets in seconds.

[![PyPI version](https://img.shields.io/pypi/v/datamedicine.svg)](https://pypi.org/project/datamedicine/)
[![Python](https://img.shields.io/pypi/pyversions/datamedicine.svg)](https://pypi.org/project/datamedicine/)
[![License](https://img.shields.io/pypi/l/datamedicine.svg)](LICENSE)

DataMedicine is a focused data-quality library for machine-learning engineers,
data scientists, analysts, Kaggle users, and AI researchers. Give it a CSV,
Parquet, Excel file, or pandas DataFrame; receive one report that explains what
is wrong, how serious it is, and how to address it.

```python
from datamedicine import validate

report = validate("train.csv", target="churn")
report.summary()
```

## Why DataMedicine exists

Dataset quality issues are routinely discovered after a model has been trained,
deployed, or shared. Generic DataFrame operations are powerful but require
knowing exactly what to look for. DataMedicine makes a quality review a normal,
repeatable part of a data workflow.

```
CSV / Parquet / Excel / DataFrame
                │
                ▼
           validate(...)
                │
     ┌──────────┼──────────┐
     ▼          ▼          ▼
 health      diagnosis   exports
     │          │          │
     └──────► treat() ◄───┘
```

DataMedicine does not hide pandas. It uses pandas efficiently, keeps the
returned cleaned data as a DataFrame, and adds a consistent quality layer above
it.

## Features

- One-line validation for CSV, Parquet, Excel, and pandas DataFrames.
- Dataset overview: shape, memory use, missingness, duplicates, schema, unique
  values, empty strings, constant columns, and file size.
- Quality checks: mixed data types, infinity, NaN, text whitespace, hidden
  Unicode, replacement characters, possible identifiers, and duplicate columns.
- ML checks: target distribution, imbalance warning, high cardinality details,
  correlation pairs, likely leakage, and possible IDs.
- Numeric statistics: min, max, mean, median, standard deviation, skewness,
  kurtosis, IQR outliers, and Z-score outliers.
- Categorical, datetime, and text-quality profiling.
- Human-readable warnings and a 0–100 health score.
- Automatic, type-aware cleaning with `autofix()` or `report.treat()`.
- HTML, JSON, Markdown, and optional PDF exports.
- Fingerprints and before/after comparisons for repeatable pipelines.

## Installation

DataMedicine requires Python 3.10 or newer.

```bash
pip install datamedicine
```

Install optional support only when needed:

```bash
pip install "datamedicine[excel]"    # XLS and XLSX
pip install "datamedicine[parquet]"  # Parquet engines
pip install "datamedicine[pdf]"      # PDF reports
pip install "datamedicine[viz]"      # matplotlib plots
```

For contributors:

```bash
git clone https://github.com/datamedicine/datamedicine.git
cd datamedicine
pip install -e ".[dev]"
pytest
ruff check .
mypy src
```

## Quick start

```python
from datamedicine import validate

report = validate("customers.csv", target="segment")
print(report.health)
report.summary()
print(report.diagnosis())
```

Expected output resembles:

```text
{'score': 91, 'status': 'Good', 'severity': 'Low'}

🏥 DataMedicine Diagnosis
=========================
Overall Health: 91/100 (Good; Low)

Detected Issues
- Missing Values — 34 cells
- Outliers — 8 IQR candidates
```

### Best practices

- Validate immediately after loading external data and again before training.
- Pass `target=` for supervised learning datasets; imbalance and numeric leakage
  checks need it.
- Treat automatic fixes as recommendations: compare before and after reports.
- Store JSON exports or fingerprints alongside model artifacts.

### Common mistakes

- Do not call `autofix(..., outliers="auto")` blindly on a domain-critical
  quantity such as medical dosage. Review clipping bounds first.
- Do not assume a health score proves a dataset is fit for a particular task.
  It reports measurable quality, not business validity or fairness.

## Validate datasets

### `validate(source, *, target=None, correlation_threshold=0.95, rare_threshold=0.01, zscore_threshold=3.0, **read_options)`

**Purpose.** Load and inspect a tabular dataset, returning a `ValidationReport`.

**Parameters.** `source` is a pandas DataFrame or a CSV, Parquet, XLS, or XLSX
path. `target` names an optional supervised-learning target. The thresholds tune
correlation, rare category, and Z-score findings. `read_options` is forwarded to
the relevant pandas reader, so `encoding="utf-8"` and `sheet_name="Data"` work.

**Returns.** A `ValidationReport`; validation does not mutate a supplied frame.

**CSV example.**

```python
from datamedicine import validate

report = validate("data/train.csv", target="label", encoding="utf-8")
```

**Excel example.**

```python
report = validate("sales.xlsx", sheet_name="January")
```

**Parquet example.**

```python
report = validate("events.parquet")
```

**DataFrame example.**

```python
import pandas as pd
from datamedicine import validate

frame = pd.DataFrame({"age": [21, None, 500], "plan": ["Free", "Pro", "Pro"]})
report = validate(frame)
```

**Performance.** Most checks are vectorized. Numeric correlation is quadratic in
the number of numeric columns, so it is the dominant operation for extremely
wide frames. Consider a narrower feature set for those inputs.

## Read the report

`ValidationReport` is the central API. It contains the source-independent data
needed for auditing and exporting, plus a private live DataFrame used by
`treat()` and certain plots.

| Property | Meaning |
| --- | --- |
| `report.health_score` | Integer quality score from 0 through 100. |
| `report.health_label` | `Excellent`, `Good`, `Fair`, `Poor`, or `Critical`. |
| `report.health` | Mapping with score, status, and action severity. |
| `report.overview` | Shape, memory, schema, duplicate, missingness, uniqueness, cardinality, and variance metrics. |
| `report.missing` | Total, percent, and missing cells by column. |
| `report.numeric` / `report.outliers` | Numeric statistics and IQR/Z-score findings. |
| `report.categorical` | Category frequencies, rare labels, dominance, and cardinality. |
| `report.datetime` | Missing, invalid, and duplicate timestamp findings. |
| `report.string_quality` | Whitespace, hidden Unicode, and encoding replacement findings. |
| `report.warnings` | Typed, human-readable warnings. |
| `report.correlations` | Numeric feature pairs above the configured threshold. |
| `report.target_distribution` | Class counts when a target was supplied. |
| `report.fingerprint` | Dataset identity, row/column counts, and schema. |

### Health score

| Score | Status | Severity |
| --- | --- | --- |
| 95–100 | Excellent | None |
| 85–94 | Good | Low |
| 70–84 | Fair | Moderate |
| 50–69 | Poor | High |
| 0–49 | Critical | Critical |

The score combines missing values, duplicate rows, detected IQR outliers, and
the number of warnings. It is designed for trend tracking and triage, not as a
replacement for domain review.

### `report.summary()`

**Purpose.** Print the concise Markdown overview.

**Parameters and return value.** None. It writes to standard output.

```python
report.summary()
```

Use it in notebooks and CI logs. For programmatic inspection, use report
properties instead of parsing console output.

### `report.diagnosis()`

**Purpose.** Produce a doctor-style quality diagnosis: overall health, detected
issues, recommended treatments, and an estimated manual triage time saving.

**Returns.** A formatted string.

```python
print(report.diagnosis())
```

The estimate is a communication aid, not a service-level guarantee.

### `report.treat(**options)`

**Purpose.** Clean the live validated DataFrame using `autofix()` options.

**Parameters.** Any `autofix()` option. With no options it removes duplicate
rows, uses type-aware missing-value treatment, chooses an outlier strategy,
trims spaces, and replaces infinity.

**Returns.** A new pandas DataFrame; the original data is not mutated.

```python
clean = report.treat(missing="auto", outliers="auto")
```

This method is unavailable on a report reconstructed from JSON because JSON does
not contain a DataFrame. Call `autofix()` directly in that case.

## Clean datasets

### `autofix(source, **options)`

**Purpose.** Apply selected, non-destructive transformations to a supported
source. It returns a pandas DataFrame.

```python
from datamedicine import autofix

clean = autofix(
    "train.csv",
    duplicates=True,
    missing="auto",
    outliers="auto",
    trim_spaces=True,
    normalize_text="lower",
    replace_infinity=True,
    convert_numeric=True,
    remove_constant_columns=True,
    remove_duplicate_columns=True,
)
```

### Autofix options

| Option | Purpose | Values |
| --- | --- | --- |
| `duplicates` | Remove repeated records. | `True` / `False` |
| `missing` | Fill missing cells. | `auto`, `mean`, `median`, `mode`, `zero`, `forward fill`, `backward fill`, or a scalar |
| `outliers` | Clip numeric outliers. | `auto`, `iqr`, `zscore` |
| `trim_spaces` | Strip surrounding text whitespace. | Boolean |
| `normalize_text` | Normalize string case. | `lower`, `upper`, `title`, or `True` for lower |
| `replace_infinity` | Convert positive/negative infinity to missing values. | Boolean |
| `convert_numeric` | Convert wholly numeric text columns. | Boolean |
| `remove_constant_columns` | Remove one-value columns. | Boolean |
| `remove_duplicate_columns` | Remove content-identical columns. | Boolean |
| `copy` | Copy an in-memory frame before cleaning. | Boolean, default `True` |

`missing="auto"` uses median for numeric values, forward fill for datetime
values, and mode for categorical/boolean values. `outliers="auto"` chooses IQR
for skewed numeric data and Z-score clipping otherwise.

**Common mistake.** `copy=False` intentionally permits mutation of a DataFrame.
Use it only when memory constraints are understood and the caller owns the data.

## Compare dataset versions

```python
before = validate("raw.csv", target="label")
clean = before.treat()
after = validate(clean, target="label")

comparison = before.compare(after)
print(comparison)
```

### `report.compare(other_report)`

**Purpose.** Quantify quality movement between two reports.

**Returns.** A `ReportComparison` with health-score, missing-value, duplicate,
and IQR-outlier deltas, plus schema changes. Negative issue deltas indicate an
improvement. Call `comparison.to_dict()` to persist it.

## Dataset fingerprints

Fingerprints detect silent dataset changes in scheduled pipelines.

```python
baseline = validate("january.csv")
latest = validate("february.csv")

print(latest.fingerprint)
changes = latest.compare_fingerprint(baseline.fingerprint)
print(changes["added_columns"])
```

The fingerprint combines schema with a stable serialization of the first 1,000 rows.
It is a practical change detector, not a cryptographic integrity proof for an
entire source file.

## Export reports

### `report.export_html(path="datamedicine-report.html")`

Writes a self-contained responsive HTML report. Open it locally or attach it to
a pull request, experiment run, or data-quality incident.

```python
report.export_html("artifacts/quality.html")
```

### `report.export_json(path="datamedicine-report.json")`

Writes a JSON report suitable for CI artifacts and dashboards. The live source
DataFrame is intentionally excluded.

```python
report.export_json("artifacts/quality.json")
```

### `report.export_markdown(path="datamedicine-report.md")`

Writes a compact Markdown summary for issues, tickets, and model cards.

```python
report.export_markdown("artifacts/quality.md")
```

### `report.export_pdf(path="datamedicine-report.pdf")`

Writes a PDF through the optional `weasyprint` dependency.

```python
report.export_pdf("artifacts/quality.pdf")
```

Install the PDF extra first. PDF export also creates the corresponding HTML
file beside the PDF as a useful diagnostic artifact.

## Visualizations

Install `datamedicine[viz]` and use the following methods. Each returns a
matplotlib `Axes`, so standard matplotlib customization remains available.

```python
report.plot_missing()
report.plot_outliers()
report.plot_correlation()
report.plot_class_distribution()  # requires validate(..., target="label")
```

If matplotlib is absent, missing/outlier/correlation plots print a clear
installation instruction and return `None`. Class-distribution plots also
require an explicit target column.

## Command line interface

### Validate

```bash
datamedicine validate train.csv --target label --html quality.html --json quality.json
```

This prints a summary and optionally writes HTML/JSON artifacts. Use shell exit
code and exported JSON in CI; do not scrape human-readable terminal output.

### Fix

```bash
datamedicine fix train.csv \
  --missing auto \
  --outliers auto \
  --duplicates \
  --trim-spaces \
  --output cleaned.csv
```

The CLI writes CSV by default, or Parquet when `--output` ends in `.parquet`.
Validate the result and compare reports before replacing a canonical dataset.

## Examples

Runnable examples live in [`examples/`](examples/). They cover CSV, Excel,
Parquet, DataFrames, all export formats, diagnosis, automatic fixing,
comparison, large datasets, imbalanced targets, missing values, and outliers.

## Performance and scalability

DataMedicine uses pandas reductions and column-level vectorized operations for
the common checks. Memory use necessarily includes the input DataFrame; file
loading is delegated to pandas. For very large data:

- Prefer Parquet over CSV when possible.
- Select model-relevant columns before validation if the source is extremely
  wide.
- Validate a representative sample for exploratory work, then the full dataset
  before release.
- Use `copy=False` only after measuring the memory and mutation trade-off.
- Correlation analysis is capped at 200 numeric columns by default. Adjust
  `max_correlation_columns` only after considering the quadratic memory cost.

## Public API reference

```python
from datamedicine import ReportComparison, ValidationReport, autofix, validate
```

`validate`, `autofix`, `ValidationReport`, and `ReportComparison` are the
stable public API for v1.0. Internal analyzer modules are implementation details
and may evolve in minor releases.

## Contributing and release quality

Run formatting, linting, typing, and tests before opening a pull request.
Changes to validation rules should include focused regression tests and explain
how they affect existing health scores. DataMedicine follows semantic versioning:
breaking changes require a major release; new backwards-compatible checks and
methods are minor releases; bug fixes are patch releases.

Maintainers should follow the repository [release checklist](RELEASE_CHECKLIST.md)
before publishing any artifact.

## License

DataMedicine is released under the [MIT License](LICENSE).