Metadata-Version: 2.4
Name: datasemver
Version: 0.8.2
Summary: Semantic versioning for datasets: diff two dataset versions and get the bump they deserve.
Author-email: Izan Vilchez <izanvilchez6@gmail.com>
Maintainer-email: Izan Vilchez <izanvilchez6@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://izanvil.github.io/datasemver/
Project-URL: Repository, https://github.com/IzanVil/datasemver
Project-URL: Issues, https://github.com/IzanVil/datasemver/issues
Keywords: semver,datasets,data-versioning,changelog,data-quality,schema-diff
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Version Control
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: pandas>=1.5
Requires-Dist: pyarrow>=23.0.1
Requires-Dist: pydantic>=2.4
Requires-Dist: pyyaml>=6
Requires-Dist: typer>=0.9
Requires-Dist: rich>=13
Provides-Extra: dev
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: sqlalchemy>=2.0; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: mypy>=1.11; extra == "dev"
Requires-Dist: types-PyYAML>=6; extra == "dev"
Requires-Dist: pyinstaller>=6.0; extra == "dev"
Requires-Dist: duckdb>=1.1; extra == "dev"
Provides-Extra: exe
Requires-Dist: pyinstaller>=6.0; extra == "exe"
Provides-Extra: sql
Requires-Dist: sqlalchemy>=2.0; extra == "sql"
Requires-Dist: psycopg2-binary>=2.9; extra == "sql"
Requires-Dist: pymysql>=1.1.1; extra == "sql"
Provides-Extra: duckdb
Requires-Dist: duckdb>=1.1; extra == "duckdb"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1.5; extra == "excel"
Provides-Extra: web
Requires-Dist: fastapi>=0.110; extra == "web"
Requires-Dist: uvicorn[standard]>=0.27; extra == "web"
Requires-Dist: python-multipart>=0.0.31; extra == "web"
Dynamic: license-file

<img src="https://raw.githubusercontent.com/IzanVil/datasemver/main/docs/assets/logo.svg" alt="" width="64" height="64">

# DataSemver

[![PyPI](https://img.shields.io/pypi/v/datasemver.svg)](https://pypi.org/project/datasemver/)
[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/downloads/)
[![Platforms](https://img.shields.io/badge/tested%20on-Linux%20%7C%20macOS%20%7C%20Windows-blue.svg)](https://github.com/IzanVil/datasemver/actions/workflows/tests.yml)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-green.svg)](https://www.apache.org/licenses/LICENSE-2.0)

**[datasemver on the web](https://izanvil.github.io/datasemver/)** — what it does and why, with the output shown rather than described.

**Your data changed. DataSemver tells you whether that is a patch, a minor or a breaking release.**

DataSemver compares two versions of a CSV, JSON or Parquet dataset, classifies every
difference it finds against a configurable rule set, and returns the semantic version bump
plus a ready-to-commit changelog entry. It is a CLI first and a Python library second, and
it needs no schema registry, no database and no service running.

## Install

```bash
pip install datasemver              # library and CLI
pip install "datasemver[web]"       # adds the dashboard
pipx install datasemver             # standalone command
```

Python 3.10 or newer. The package ships typed, so `py.typed` annotations reach type
checkers.

## Use it

```bash
datasemver diff old.csv new.csv --current-version 1.4.2
```

```
╭───────────── DataSemver ──────────────╮
│ Suggested bump: MAJOR                 │
│ 0.0.0 -> 1.0.0                        │
│                                       │
│ old: old.csv (8 rows)                 │
│ new: new.csv (10 rows)                │
╰───────────────────────────────────────╯
                                    Columns
┏━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ column      ┃ status    ┃ type old ┃ type new ┃ nulls         ┃ cardinality ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ country     │ added     │ -        │ string   │ - -> 0.0%     │ - -> 4      │
│ phone       │ modified  │ int64    │ string   │ 0.0% -> 0.0%  │ 8 -> 10     │
│ email       │ modified  │ string   │ string   │ 25.0% -> 0.0% │ 6 -> 10     │
│ legacy_code │ removed   │ string   │ -        │ 0.0% -> -     │ 8 -> -      │
└─────────────┴───────────┴──────────┴──────────┴───────────────┴─────────────┘
                                        Changes
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ severity ┃ rule                      ┃ description                                   ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ MAJOR    │ column_removed            │ Column 'legacy_code' was removed              │
│ MAJOR    │ type_changed_incompatible │ Column 'phone' changed type from int64 to     │
│          │                           │ string                                        │
│ MINOR    │ row_count_increased       │ Row count grew from 8 to 10 (+25.00%)         │
│ MINOR    │ column_added              │ Column 'country' was added                    │
│ PATCH    │ nulls_fixed               │ Column 'email' nulls dropped from 25.0% to    │
│          │                           │ 0.0%                                          │
└──────────┴───────────────────────────┴───────────────────────────────────────────────┘
```

A removed column and an `int64` that became a `string` make this a breaking release.
Without `--output`, the changelog entry is printed at the end; with it, the entry is
prepended to the file you name.

```bash
datasemver diff old.csv new.csv --output CHANGELOG.md
datasemver diff old.csv new.csv --json | jq -r '.bump'
datasemver rules examples/lenient_rules.yaml
```

| Option | Short | Description |
| --- | --- | --- |
| `--rules PATH` | `-r` | Rule file replacing the bundled defaults |
| `--current-version TEXT` | `-c` | Version the new dataset is bumped from (default `0.0.0`) |
| `--output PATH` | `-o` | Write the changelog entry, prepending it if the file exists |
| `--json` | | Print the full report as JSON instead of the tables |

## What it looks at

- **Schema** — columns added, removed and renamed, dtype changes, nullability.
- **Content** — row counts, cardinality, mean and standard deviation of numeric columns,
  mode and category sets of categorical ones.
- **Semantics** — renames inferred from the similarity of both the column name and its
  values, so `user_name` becoming `username` is one rename rather than a removal plus an
  addition.

The bump is the strongest severity across every classified change. Changes no rule covers
are reported as unclassified and never inflate it.

| Bump | Meaning for consumers |
| --- | --- |
| **Major** | Existing queries and pipelines can break |
| **Minor** | New information, existing contracts still hold |
| **Patch** | Same meaning, better data |

## Formats

Detected by extension: `.csv`, `.csv.gz`, `.tsv`, `.tsv.gz`, `.json`, `.jsonl`, `.ndjson`, `.parquet`, `.pq`,
`.feather`, `.arrow`, `.xlsx` and `.xlsm`.

The delimiter of a `.csv` is detected from its first lines — comma, semicolon, tab and pipe
are recognised, and a character that only appears inside quoted values does not win — while
`.tsv` always uses the tab. Set `DATASEMVER_CSV_DELIMITER` to skip detection and force one
character, the tab written as `\t`.

Nested JSON objects and the structs of the Arrow-backed formats are flattened with a `.`, so
`{"user": {"name": "..."}}` is profiled as `user.name`. Types are inferred for the text
formats; Parquet and Feather carry their own schema and are trusted as they stand. A workbook
names its sheet after `#`, as `quarterly.xlsx#Q3`.

## Engines

Every statistic in a profile is an aggregate, and an aggregate does not need the dataset in
memory. With `pip install "datasemver[duckdb]"`, `--engine duckdb` computes them over the file
instead of over a dataframe — the same numbers for about half the memory — and
`--engine duckdb-sketch` estimates the quantile grid for a third of that again. Measured on 16M
rows: 21.4 s and 3.2 GB becomes 16.1 s and 1.7 GB, or 3.1 s and 1.2 GB.

## Databases

A source can be a table instead of a file, with the connection URL naming the database and
the fragment naming the table:

```bash
pip install "datasemver[sql]"

datasemver diff "sqlite:///snapshots.db#customers_v1" "sqlite:///snapshots.db#customers_v2"
datasemver diff "postgresql://reader:secret@warehouse:5432/analytics#customers" new.csv
```

SQLite needs no driver; PostgreSQL and MySQL use the ones the `sql` extra installs. Quote the
argument, since `#` opens a comment in most shells. Passwords are removed before the source
reaches a report. Whole tables only for now: no views, no queries, no schema qualification.

## Rules

Every severity is a list of rules, evaluated `major`, then `minor`, then `patch`. The first
rule that matches a change assigns its severity.

```yaml
major:
  - column_removed
  - type_changed_incompatible
  - row_count_decrease_greater_than: 20

minor:
  - column_added
  - row_count_decreased

patch:
  - nulls_fixed
  - minor_stat_change
```

Pass it with `--rules custom.yaml`, and check how it was parsed with `datasemver rules
custom.yaml`. Threshold rules pair with their plain counterpart in a lower severity, which
then acts as the fallback. Unknown rule names and severities are errors, not silent no-ops.

## Python API

```python
from datasemver import analyze

report = analyze("old.csv", "new.csv", current_version="1.4.2")

print(report.bump)          # Severity.MAJOR
print(report.next_version)  # 2.0.0

for item in report.classified:
    print(item.severity, item.rule, item.change.description)
```

`analyze_schemas()` takes two already loaded profiles, so dataframes from anywhere can be
compared without touching the filesystem:

```python
import pandas as pd
from datasemver.core.analyzer import analyze_schemas
from datasemver.formats.loader import schema_from_frame

report = analyze_schemas(
    schema_from_frame(pd.read_sql(query, engine), "warehouse@yesterday"),
    schema_from_frame(pd.read_sql(query, engine), "warehouse@today"),
)
```

## Also in the box

- A **web dashboard** — FastAPI backend, no-build frontend — under the `web` extra, run
  with `uvicorn datasemver_web.backend.main:app`. It is a local tool with no
  authentication: keep it on the loopback interface.
- A **GitHub Action** that analyses the datasets a pull request touches and posts the
  suggested bump as a comment, rewritten on each push.
- Two ready-made rule profiles, strict and lenient, and a full catalogue of rules, metrics
  and thresholds.

Those, the source, the changelog and a Spanish edition of this page live in the project
repository, linked from this page's sidebar.

## Security

Reading a dataset parses it. CSV and JSON go through pandas and the standard library, which
do not execute file content; Parquet goes through pyarrow, and the dependency floor is
`pyarrow>=23.0.1` because earlier versions carried a critical code-execution flaw
(CVE-2023-47248) triggered by a malicious Parquet file. Do not lower that floor. Rule files
are YAML loaded with `yaml.safe_load` and cannot execute code. The library and the CLI never
open a socket, and write nothing unless you pass `--output`.

## License

Apache License 2.0. The licence grants no right to the DataSemver name or marks. Releases up to and including 0.7.0 were published under the MIT licence and remain available under it.
