Metadata-Version: 2.4
Name: patchbucket
Version: 0.1.0
Summary: Row-level replication between machines whose only shared resource is an object store
Author: the patchbucket contributors
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/thefishplate/patchbucket
Project-URL: Source, https://github.com/thefishplate/patchbucket
Project-URL: Changelog, https://github.com/thefishplate/patchbucket/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/thefishplate/patchbucket/issues
Keywords: sqlite,replication,sync,rclone,object-storage,offline-first
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Database
Classifier: Topic :: System :: Archiving :: Mirroring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Dynamic: license-file

# patchbucket

Append-only patch-log replication for SQLite over an object store.

Each machine owns a numbered, immutable log of gzip-JSON row patches. Peers pull
unapplied patches and apply them in sequence order, idempotently. No server, no
daemon, no third-party Python dependencies — rclone is the only external runtime
dependency, and it is called as a subprocess.

```
remote_root/
  central_manifest.json          ← who exists, and each machine's head
  machines/
    laptop/   patch-000001.json.gz   ← immutable, write-once
              patch-000002.json.gz
    desktop/  patch-000001.json.gz
```

## What this is, precisely

Row-level replication between machines whose only shared resource is an object
store — no server, no daemon, no node reachable from another, and no
third-party Python dependencies.

The local store sits behind an interface. **SQLite is the only
implementation**, and the interface is unproven until a second one exists, so
read every claim below as being about SQLite.

Every element of that is prior art. The combination is the part that appears
unserved, and it is the only claim made here.

## Prior art

This exists because of a specific gap, not because the field is empty. If your
situation is described in the right-hand column below, use that instead — those
projects are more mature, more reviewed and better supported than this one.

| If you need | Use |
|---|---|
| Continuous backup and point-in-time restore, one writer | [Litestream](https://litestream.io) — WAL streaming to S3-compatible storage |
| Automatic row-level conflict resolution | [cr-sqlite](https://github.com/vlcn-io/cr-sqlite) — CRDT tables. Gives you merge semantics; you still supply the transport |
| An efficient one-off or scheduled copy between two hosts | [`sqlite3-rsync`](https://sqlite.org/rsync.html) — official, page-level, consistent snapshot over SSH |
| Multi-master replication with nodes that can reach each other | [Marmot](https://github.com/maxpert/marmot) (NATS), [rqlite](https://rqlite.io) (Raft) |
| A managed local-first platform | Turso / libSQL embedded replicas, and the wider local-first ecosystem |

What none of them do is replicate between machines whose **only** shared
resource is a dumb object store — no server to run, no port to open, no node
reachable from another, and no compiled extension to install. That is the case
this addresses: several intermittently-connected machines, a Google Drive or S3
bucket between them, and nothing else.

The common alternative in that situation is to put the `.db` file in a synced
folder. That corrupts under concurrent writers, and avoiding it is the whole
point of what follows.

### Deliberate limits

- **Not a CRDT.** Conflict policy is yours to write, in two methods. The
  library handles identity, sequencing, transport and idempotent application.
- **Not continuous.** Replication happens when you call it. There is no daemon
  and nothing runs in the background.
- **Not a consistency guarantee across writers.** Each machine owns its own
  append-only log; convergence is eventual and depends on peers pulling.
- **Not audited.** No third-party review has been done.
- **Not store-agnostic in practice.** The store interface exists and the
  manager holds no SQL, but SQLite is the only implementation and nothing has
  tested the seam from the other side.

### On backends

The default head-discovery strategy derives each peer's position from the patch
files present, because that is the only approach that is safe on a backend
offering no compare-and-swap — Google Drive via rclone being the case in point.
Backends that do offer conditional writes (S3 with ETags) can support a cheaper
strategy safely; see `FINDINGS.md` §4. A strategy that is safe on one backend
and silently lossy on another should never be selectable by configuration
alone.

## Usage

Subclass and implement two methods:

```python
from patchbucket import PatchManager

class MyPatchManager(PatchManager):
    def rows_export_since(self, conn, watermark):
        ...   # return {"items": [...], "exported_at": <unix float>}

    def row_patch_apply(self, conn, patch):
        ...   # apply that dict to your schema

pm = MyPatchManager(db_path, patch_dir, remote_root="gdrive:myapp")
pm.sync_all()          # push local changes, pull and apply peers'
```

Machine identity, sequencing, immutable upload, manifest handling, peer
discovery and idempotent application are all handled by the base class.

## Tests

```powershell
python patchbucket_dev.py            # 145 tests, the default command
python -m unittest patchbucket_dev   # or the standard entry point
```

145 tests, ~17 s, no network and no rclone binary required — the suite uses a
`LocalRemote` fake and a `TestPatchManager` subclass. It covers convergence
across two and three machines, idempotent re-application, immutable-overwrite
conflicts, corrupt downloads, unsupported schema versions, and resumption after
a failed apply.

## Status

**Extracted 2026-08-26; not yet adopted by any consumer.**

The intended consumers are three private projects. Two of them each carried
their own implementation of this idea, and those two had already diverged into
incompatible remote layouts — which is why this repo exists. Nothing has been
migrated yet.

**The wire format is not settled.** See `FINDINGS.md`. Measurement favours a
different layout from the one implemented here, so the remote structure above
should be treated as provisional.

## Origin

Extracted from a domain-free patch manager in a private project, itself a
rewrite of sync code in an earlier one. Copied here unchanged — the first
commit is a pure extraction with the test suite intact, so any behaviour change
is a later, reviewable commit.
