Metadata-Version: 2.4
Name: tether-vcs
Version: 0.1.0b1
Summary: jj-style version control for heterogeneous datasets (files/object stores, Icechunk, Neon, Iceberg, Delta, Lance, lakeFS, DuckLake, Dolt, git)
Keywords: version-control,data-versioning,icechunk,iceberg,neon,delta-lake,lance,lakefs,ducklake,dolt,jujutsu
Author: tether contributors
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Version Control
Classifier: Typing :: Typed
Requires-Dist: tomlkit>=0.12
Requires-Dist: typer>=0.12 ; extra == 'all'
Requires-Dist: obstore>=0.11 ; extra == 'all'
Requires-Dist: icechunk>=1.0 ; extra == 'all'
Requires-Dist: httpx>=0.27 ; extra == 'all'
Requires-Dist: psycopg[binary]>=3.1 ; extra == 'all'
Requires-Dist: pyiceberg>=0.10 ; extra == 'all'
Requires-Dist: pyarrow>=15 ; extra == 'all'
Requires-Dist: deltalake>=1.0 ; extra == 'all'
Requires-Dist: pylance>=1.0 ; extra == 'all'
Requires-Dist: lakefs>=0.8 ; extra == 'all'
Requires-Dist: duckdb>=1.3 ; extra == 'all'
Requires-Dist: pymysql>=1.1 ; extra == 'all'
Requires-Dist: obstore>=0.11 ; extra == 'azure'
Requires-Dist: typer>=0.12 ; extra == 'cli'
Requires-Dist: deltalake>=1.0 ; extra == 'delta'
Requires-Dist: pymysql>=1.1 ; extra == 'dolt'
Requires-Dist: duckdb>=1.3 ; extra == 'ducklake'
Requires-Dist: obstore>=0.11 ; extra == 'gcs'
Requires-Dist: pyiceberg>=0.10 ; extra == 'iceberg'
Requires-Dist: pyarrow>=15 ; extra == 'iceberg'
Requires-Dist: icechunk>=1.0 ; extra == 'icechunk'
Requires-Dist: lakefs>=0.8 ; extra == 'lakefs'
Requires-Dist: pylance>=1.0 ; extra == 'lance'
Requires-Dist: httpx>=0.27 ; extra == 'neon'
Requires-Dist: psycopg[binary]>=3.1 ; extra == 'neon'
Requires-Dist: obstore>=0.11 ; extra == 'objectstore'
Requires-Dist: psycopg[binary]>=3.1 ; extra == 'postgres'
Requires-Dist: obstore>=0.11 ; extra == 's3'
Requires-Python: >=3.11
Project-URL: Homepage, https://evanlyall.com/tether/
Project-URL: Documentation, https://evanlyall.com/tether/
Project-URL: Repository, https://github.com/elyall/tether
Project-URL: Changelog, https://github.com/elyall/tether/blob/main/CHANGELOG.md
Provides-Extra: all
Provides-Extra: azure
Provides-Extra: cli
Provides-Extra: delta
Provides-Extra: dolt
Provides-Extra: ducklake
Provides-Extra: gcs
Provides-Extra: iceberg
Provides-Extra: icechunk
Provides-Extra: lakefs
Provides-Extra: lance
Provides-Extra: neon
Provides-Extra: objectstore
Provides-Extra: postgres
Provides-Extra: s3
Description-Content-Type: text/markdown

# tether

[![PyPI](https://img.shields.io/pypi/v/tether-vcs)](https://pypi.org/project/tether-vcs/)
[![Python](https://img.shields.io/pypi/pyversions/tether-vcs)](https://pypi.org/project/tether-vcs/)
[![CI](https://github.com/elyall/tether/actions/workflows/ci.yml/badge.svg)](https://github.com/elyall/tether/actions/workflows/ci.yml)
[![Coverage](https://codecov.io/gh/elyall/tether/graph/badge.svg)](https://codecov.io/gh/elyall/tether)
[![Docs](https://github.com/elyall/tether/actions/workflows/docs.yml/badge.svg)](https://evanlyall.com/tether/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/elyall/tether/blob/main/LICENSE)

**Version control (fingerprints, pins, & forks) for heterogeneous datasets.**

> Status: alpha. See the [changelog](https://github.com/elyall/tether/blob/main/CHANGELOG.md).
>
> State: vibe coded with Claude Fable 5.1. **USE AT YOUR OWN RISK.**

Think of `tether` as **DVC for *branchable* systems**: like DVC it commits small
manifests into your git/jj repo, but where DVC only fingerprints files, tether
also *pins* and *forks* live systems. Every backend is fingerprinted; what
else tether can do with each depends on what the system offers:

| Backend | Recover | Pin | Fork | Promote | Merge | Diff | History |
| --- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| local files and directories; object-store prefixes (S3, GCS, Azure) | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
| single object-store objects with versioning enabled | 🟡 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| git / [jj](https://jj-vcs.dev) repositories | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Icechunk](https://icechunk.io) repositories | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| [Neon](https://neon.com) Postgres databases | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| [Apache Iceberg](https://iceberg.apache.org) tables | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| [Delta Lake](https://delta.io) tables | 🟡 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| [Lance](https://lance.org) datasets | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ |
| [lakeFS](https://lakefs.io) repositories *(experimental)* | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [DuckLake](https://ducklake.select) catalogs *(experimental)* | 🟡 | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| [Dolt](https://www.dolthub.com) databases *(experimental)* | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

**Recover**: can a committed state be opened again later? ✅ held by a pin;
🟡 reopenable by version id for as long as the system keeps that version
(object versioning, Delta/DuckLake snapshots) -- tether records it but has no
ref of its own to hold it; ❌ tether can only tell you it changed. **Pin**: a
durable native ref (tag, protected branch) tether creates at commit time.
**Fork**: a writable branch off a pin for `tether new`. **Promote**:
fast-forward the base branch to the fork; **Merge**: a native three-way merge
when it is not a fast-forward. **Diff**: `tether diff --content` describes
what changed inside the object. **History**: `tether log` lists the object's
own snapshots/versions/commits. The full matrix -- state fields, flags,
per-backend caveats -- is in the
[backends guide](https://evanlyall.com/tether/user-guide/backends.html).
*Experimental* 🧪: `tether add --create` makes a store tether then owns
(Icechunk, git, a local directory) and `tether gc --delete-stores` removes it
once nothing references it -- see
[reclaiming storage](https://evanlyall.com/tether/user-guide/reclaiming-storage.html).
What graduates, what is tested against real services, and what goes before
0.1.0 is in [ROADMAP.md](https://github.com/elyall/tether/blob/main/ROADMAP.md).

Documentation: <https://evanlyall.com/tether/> --
[getting started](https://evanlyall.com/tether/user-guide/getting-started.html),
[concepts](https://evanlyall.com/tether/user-guide/concepts.html),
[pinning](https://evanlyall.com/tether/user-guide/pinning.html),
[branching and writing](https://evanlyall.com/tether/user-guide/branching-and-writing.html),
[reclaiming storage](https://evanlyall.com/tether/user-guide/reclaiming-storage.html),
[use cases](https://evanlyall.com/tether/user-guide/use-cases.html),
[CLI](https://evanlyall.com/tether/user-guide/cli.html),
[configuration](https://evanlyall.com/tether/user-guide/configuration.html),
[backends](https://evanlyall.com/tether/user-guide/backends.html),
[registries and SQL](https://evanlyall.com/tether/user-guide/registries-and-sql.html),
[writing a backend](https://evanlyall.com/tether/user-guide/extending.html),
[caveats and performance](https://evanlyall.com/tether/user-guide/caveats-and-performance.html),
plus the generated API and CLI reference.

## The model

A dataset is a git/jj repository with one small manifest per object, and its
bookmarks and the stores' branches are one shape:

- **Commits hold references.** Each dataset commit records, per object, an
  exact state and -- where the system allows it -- a *pin*: a native,
  GC-proof ref (an Icechunk tag, a protected Neon branch, a git tag) that
  holds that state. A commit is a complete, reproducible position for every
  object at once; `tether open KEY --rev C` reads it.
- **Bookmarks hold branches.** The trunk bookmark (`main`) stands for every
  object's upstream branch: working on it writes there. Any other bookmark
  stands for one branch per system, named after it (`tether.ws.<dataset>.feature`),
  forked from the pins of the commit it started at -- decided by
  `tether new -b feature`, created on the first write. A working copy on no
  bookmark is read-only.

`commit` pins the heads of your bookmark's branches and moves the bookmark to
the new commit, the way a git branch follows its commits. `promote` lands a
bookmark on the trunk: each system's upstream branch fast-forwards (or
merges) to the bookmark's branch, then `main` moves to the bookmark's commit.
`pull` is the fetch: on `main` it reads every object's upstream branch (and,
for systems without branches, the object itself) and commits what moved.

Everything history-shaped (commits, bookmarks, workspaces, sharing, undo of
the manifests) is the VCS's. What the VCS cannot see -- what tether did to
the *stores* -- is in tether's own operation log (`tether ops`, `tether
undo`).

## Why this exists (prior art)

Every existing tool versions a single layer:

| Tool | Scope | Relationship to tether |
| --- | --- | --- |
| lakeFS, Quilt, Oxen, DataChain | objects / files | analog of our `file` objects; lakeFS is also a backend |
| **DVC** (lakeFS-owned) | files/objects in git | closest structural analog; no pin/fork of live systems |
| Dolt, pgGit, Neon, Databricks Lakebase | one database | vendor-bound; no external-object pins; Dolt and Neon are backends |
| Nessie, Bauplan | Iceberg catalog branching | Iceberg-only |
| Icechunk, Lance, Delta, DuckLake | one dataset / table / catalog | we use them as backends |
| [Yggdrasil](https://github.com/replikativ/yggdrasil) (replikativ) | cross-system: Clojure protocol stack (snapshot / branch / merge / watch) over Git, ZFS, Btrfs, IPFS, Iceberg, Datahike, lakeFS, Dolt, Podman, with an HLC-coordinated workspace | the closest conceptual sibling; not adoptable from Python (JVM library; its own README marks the Python binding as unmaintained since the initial release) |
| Dagster observable assets | staleness detection | analog of our snapshot/drift step; no branching |

Nothing provides unified version control *across* files + Icechunk + Postgres +
Iceberg with pinning and forking. So tether borrows jj's working-copy and
operation-log ideas (`new REV`, stale-working-copy detection, `undo`), DVC's
manifests-in-VCS layout, and [Yggdrasil](https://github.com/replikativ/yggdrasil)'s
observe-then-record shape (a workspace that watches independent systems and
records their snapshots, rather than a store that holds the data).

## Install

```bash
pip install tether-vcs[cli]                 # core + CLI
pip install tether-vcs[cli,icechunk,neon]   # add backends you need
pip install tether-vcs[all]                 # everything
```

Extras: `cli`, `objectstore` (S3/GCS/Azure for `file`; `s3`/`gcs`/`azure` are
aliases), `icechunk`, `neon`, `iceberg`, `delta`, `lance`, `lakefs`, `ducklake`,
`dolt`, `postgres` (`tether publish` / `import` against Postgres), `all`.
`git`/`jj` must be on `PATH`.

## Quickstart

tether lives *inside* a git or jj repository and commits small manifests
there; that repository's history, branches, and workspaces are the dataset's
too. Nothing is contacted until you ask for a state.

```bash
jj git init my-dataset && cd my-dataset      # or: git init my-dataset
tether init
tether add zarr/imaging --kind icechunk s3://bucket/imaging.icechunk
tether add db/metrics   --kind neon --project-id prj-123 --database neondb --role runner
tether add raw/plate1   --kind file s3://bucket/raw/plate1/

tether status                       # clean / modified / drifted per object (local; --snapshot re-fingerprints)
tether commit -m "Baseline"         # on main: pin each object's upstream branch, write the manifests, jj/git commit
tether new -b relabel               # a bookmark: one branch per system, named after it, forked on first write
tether open db/metrics              # postgresql://... on the relabel branch
tether commit -m "Relabel plate1"   # pins the branches' heads and moves the bookmark
tether promote                      # land it: each system's main fast-forwards (or merges), then main moves to the commit
tether verify --all-history         # every pin any commit ever named still resolves
```

```python
from tether import Repo

repo = Repo.find(".")
h = repo.open("zarr/imaging")  # writable IcechunkHandle on this bookmark's branch
ro = repo.open(
    "zarr/imaging", rev="main"
)  # read-only at main's pin; TETHER_REV=<rev> makes this the default
```

Every command that writes to a store takes `--dry-run` (and `--plan FILE` /
`--from-plan FILE`) so the writes can be reviewed first. The
[getting started guide](https://evanlyall.com/tether/user-guide/getting-started.html)
walks through this with output; the
[CLI guide](https://evanlyall.com/tether/user-guide/cli.html) has every
command.

## jj or tether?

A tether command exists where an operation has two halves -- one in the VCS,
one in the stores -- that must happen together: `commit` (pin, commit, move
the bookmark), `new` (create or join a bookmark, then its branches), `pull`,
`promote`, `restore`, `abandon`, `forget-workspace`, and `undo` / `repair`
for what tether itself did. Everything that only touches files and history --
describe, squash, rebase, push, `jj undo` of a non-tether operation -- is the
VCS's, and tether notices what it needs to: a bookmark deleted, renamed, or
moved by hand shows up in `status` with what to do, as does a vanished dataset
commit. The
[concepts guide](https://evanlyall.com/tether/user-guide/concepts.html#jj-or-tether)
has the table.

## Non-goals

- Sitting in the data path: `open` hands back the system's native handle and
  steps aside.
- Running between commands: no daemon, no watcher; states are compared when
  a command asks.
- Reimplementing history, branching, or sharing: that is the VCS's job.
  `tether ops` / `undo` / `repair` cover only what tether did to the stores,
  which the VCS cannot see.
- Cross-system transactions or a query layer.

## Development

```bash
uv sync --all-extras
uv run ruff format --check . && uv run ruff check .
uv run ty check
uv run pytest
```

The `publish` / `import` tests start an ephemeral PostgreSQL cluster through
`pytest-postgresql`; they need `pg_ctl` on `PATH` or a Homebrew / Debian
install (`brew install postgresql@16`; GitHub's Ubuntu runners ship it) and
skip otherwise.

## License

Apache-2.0. See [LICENSE](LICENSE).
