Metadata-Version: 2.5
Name: peekdd
Version: 0.1.0
Summary: Write an original onto a block device, rewriting only the blocks that differ
Project-URL: Homepage, https://gitlab.com/paulto/peekdd
Project-URL: Repository, https://gitlab.com/paulto/peekdd
Project-URL: Issues, https://gitlab.com/paulto/peekdd/-/issues
Author-email: Paul Tobias <gitlab@tobias.pt>
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Keywords: block device,dd,delta,flash,sd card,wear
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Hardware
Classifier: Topic :: Utilities
Requires-Python: >=3.14
Description-Content-Type: text/markdown

# peekdd

Writes an original (file, block device, or stdin) onto a target, reading both sides and rewriting only the blocks that differ. Re-flashing an SD card with a rebuilt image of the same distribution then costs a full read but almost no writes: most of the card is already correct, so it survives more re-flashes and each one finishes in the time the read takes rather than the time the write takes.

```
peekdd [--block-size SIZE] [--buffer SIZE] [--dry-run] ORIGINAL TARGET
```

## Install and run

peekdd needs write access to the target, which for a block device means root:

```
uv tool install peekdd
sudo $(command -v peekdd) image.img /dev/mmcblk0
```

Straight from a checkout, without installing anything:

```
sudo python3 peekdd.py image.img /dev/mmcblk0
xzcat image.img.xz | sudo python3 peekdd.py - /dev/mmcblk0
```

## Block size

The block is the unit of comparison: a block is either identical on both sides or rewritten whole. Adjacent differing blocks are coalesced into one write, so the block size sets how much identical data gets rewritten around a change, never how many write calls happen.

The default is the target disk's `queue/discard_granularity` from sysfs, with a 4 KiB floor (the largest common logical sector and the flash page size). On SD and eMMC that number is the card's erase unit, so writes land as whole allocation-unit programs with no read-modify-write inside the card: detection is exact. NVMe and SATA SSDs never publish their NAND erase block, so there the value is TRIM granularity and the floor wins. `--block-size` overrides it and takes any positive multiple of 512, written as a byte count (`4096`) or with a binary suffix (`4K`, `4M`, `4MiB`, case-insensitive). The `KB`, `MB`, `GB` and `TB` spellings are refused rather than read as binary, because `dd`, `truncate` and `losetup` all read them as powers of 1000.

## Read-ahead

Each side is read by its own thread and a third one writes, so a stall on one side no longer stalls the other and the buffers in between absorb the difference. `--buffer` sets how much memory those buffers take, split evenly between the original and the target side for now. It takes the same sizes as `--block-size`, any positive multiple of 512, and is refused when it cannot hold two chunks per side — a chunk being the internal IO unit, 4 MiB rounded up to a whole number of blocks. The default is 64 MiB: eight chunks a side at 4 MiB chunks, fewer as a larger block size grows the chunk, and raised past 64 MiB only when four chunks no longer fit in it, which takes a block size above 16 MiB.

## Dry run

`--dry-run` compares everything and writes nothing, reporting what it would have rewritten. It doubles as verification: run it after a transfer and zero rewritten bytes proves the target matches the original. There is no separate `--verify` pass, because with a stdin original there is nothing to re-read — rerun with the same pipe instead.

## Progress and interruption

While stderr is a terminal a pv-style line is redrawn in place there; redirect stderr and only the summary is left, and stdout stays empty either way.

The meter on that line is the pipeline: `[++==+www|?ttt....]`, the cells left of the `|` the chunks already compared, the cells right of it the chunks ahead of the compare, with `.` unread, `o` read from the original, `t` read from the target, `?` read on both sides and waiting for the compare, `=` equal, `w` differing with the write pending, `+` written, which is also what a dry run marks a chunk it would have rewritten. The first frames have blanks on the left, there being no history yet.

Reading it: `[wwwwwwww|ttttttt.]` is every original buffer held by a pending write, so the writes are the limit; `[++==+=+=|?ttttttt]` is the target far ahead with nothing pending, so the original — a slow `curl`, say — is the limit; `[+======+|?ooooooo]` is the reverse, the target reads are the limit; `[========|........]` is both readers starving the compare. A lone `w` far behind a long equal run holds one buffer and limits nothing, and neither do the three in the legend above: at eight chunks a side it takes eight pending writes to stall the original reader.

The summary says the same in numbers: `peekdd: waited on original 40%, on target reads 5%, on target writes 55%` is the share of the run the compare spent waiting for its next original chunk, for its next target chunk, and for the writer to hand back an original buffer. It waits on one of the three at a time, so they never overlap and add up to at most 100.

Ctrl-C exits 130 at once as far as peekdd is concerned: the chunks still queued for the writer are dropped and the card is not flushed. The kernel writes back what was already written, and on a block device the last close waits for that, the same wait as after a killed `dd`, so the prompt comes back once the device has caught up — usually a few chunks' worth, each chunk's writeback having been started as it was written. Eject or `sync` the card the usual way before pulling it. An interrupt in mid-write can leave a partial block, which the next run's compare repairs. Rerun after an interrupt and only the blocks still differing are written, which is all resuming means here: the compare comes first every time, so a rerun costs another full read and the writes that are still missing.

## Caveats

An original larger than the target is refused before anything is written, except from stdin: a pipe has no size, so the run fails at the overflow with the target already partially updated, the same way `dd` ends. An original smaller than the target leaves the trailing target bytes untouched and reports how many.

Target reads and writes ask for `RWF_DONTCACHE`, so they neither evict what was cached before nor leave the page cache full of the transfer. Block devices accept the flag from Linux 7.3; older kernels answer `ENOTSUP`, while an interpreter built without `preadv2` raises `NotImplementedError` instead — the uv-managed free-threaded 3.14 this was developed against does exactly that. Either way peekdd falls back to `posix_fadvise(DONTNEED)` after each chunk, which prunes the reads and cannot drop pages still dirty from the writes, but starts their writeback, which keeps the dirty backlog to a few chunks.

## Left for later

Splitting `--buffer` between the two sides on demand instead of evenly, and allocating its slots lazily; partition targets whose start is not aligned to the erase unit; `BLKDISCARD` instead of writing zeros for zero-filled blocks.

## How it was written

With an LLM: the design was settled in an interview with Claude Code (the record is in `docs/plans/`), and the code, tests and this README were written and reviewed by Claude agents working from that record. The author made the design decisions, ran the result on real cards and reviewed the changes.

## License

AGPL-3.0-or-later. See [LICENSE](https://gitlab.com/paulto/peekdd/-/blob/master/LICENSE).
