Metadata-Version: 2.4
Name: physis-lm
Version: 0.2.4
Summary: Reference PyTorch implementation of the Physis-LM preprint (byte-native, non-autoregressive, hierarchical-hourglass language model). Made with AI assistance.
Author: Omur Bera Isik
License: Apache-2.0
Keywords: language-model,non-autoregressive,byte-level,hierarchical,pytorch,research
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.24
Provides-Extra: torch
Requires-Dist: torch>=2.1; extra == "torch"
Provides-Extra: jax
Requires-Dist: jax>=0.4; extra == "jax"
Requires-Dist: jaxlib>=0.4; extra == "jax"
Provides-Extra: tf
Requires-Dist: tensorflow>=2.16; extra == "tf"
Provides-Extra: all
Requires-Dist: torch>=2.1; extra == "all"
Requires-Dist: jax>=0.4; extra == "all"
Requires-Dist: jaxlib>=0.4; extra == "all"
Requires-Dist: tensorflow>=2.16; extra == "all"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: hypothesis>=6.0; extra == "test"
Requires-Dist: torch>=2.1; extra == "test"
Provides-Extra: hf
Requires-Dist: transformers>=4.40; extra == "hf"
Dynamic: license-file

# physis-lm

Reference PyTorch implementation of the preprint **"Physis-LM: A Parallel
Non-Autoregressive Language Model with Hierarchical Latent Compression and
Toward Guaranteed Output Consistency"** (Omur Bera Isik, 2026) — a byte-native,
non-autoregressive language model built around a Dynamic-Depth Hierarchical
Hourglass (DDHH), a Physis Long-Range Bridge (PLRB), Length Parameters (LP), and
a Soft Consensus Module (SConM).

Made with AI assistance. Version 0.2.4.

## Installation

Physis-LM's three numeric backends are **optional** -- install only the framework you use:

```
pip install physis-lm[torch]   # PyTorch: training + reference inference (most users)
pip install physis-lm[jax]     # JAX forward/generation parity backend
pip install physis-lm[tf]      # TensorFlow forward/generation parity backend
pip install physis-lm[all]     # all three
```

`import physis_lm` itself pulls in no framework. If you touch a backend whose framework
isn't installed, you get a clear error naming the exact extra to add -- not a cryptic
`ModuleNotFoundError`. Training and the reference inference path require the `[torch]`
extra. (Note DEP1.)

## Streaming from Hugging Face (WebDataset) & multimodal bytes (0.1.6)

Physis-LM is a byte model, so it trains on any file as a byte stream, and it can stream
WebDataset tar shards (the format Hugging Face serves for large corpora) without unpacking:

```python
from physis_lm.webdata import WebDatasetByteStream

# local shards or a glob; select which member types to feed (text-only shown)
wds = WebDatasetByteStream("shards/train-{0000..0100}.tar", select={"txt"})
wds = wds.examples(context_len=112, Nmax=8, pad_byte=0)   # -> yields (ctx, tgt) tensors

# or straight from a Hugging Face streaming dataset:
#   from datasets import load_dataset
#   hf = load_dataset("some/webdataset-corpus", streaming=True, split="train")
#   wds = WebDatasetByteStream.from_hf(hf, select={"txt"}).examples(112, 8, 0)
```

It is **streaming and memory-bounded** (only the current shard member is held). Any file type
is accepted as bytes; `modality_tags=True` prepends a stream-type marker, and
`data.read_document_bytes(..., expand_archives=True)` turns a ZIP into its member documents.

**Decoding compressed containers (`decode_containers=True`).** Raw PNG/ZIP/gzip bytes are
already compressed, hence near-random and unlearnable (study S20). The structure lives in the
*decoded* content, so the loader can decode before feeding the model: images -> raw pixels,
single-stream archives (gzip/zlib/bz2/xz) -> decompressed contents. Measured (study S21):
decoded image pixels beat their byte-entropy floor by ~0.4 bits and decompressed text is
learnable, where the raw containers were not. Pillow is optional (`physis-lm[image]`); without
it, image bytes pass through unchanged.

**Honest scope:** decoding gives the model learnable *bytes* (pixels, decompressed text). It
does not add a vision/audio model or cross-modal understanding -- it removes the
compressed-container barrier so the model can learn the structure that is actually there.

## Status & honest scope

This is a **small-scale research implementation**. On held-out Shakespeare at 2.77M parameters (CPU), the best configuration reaches 4.21 bits/byte teacher-forced and 4.70 self-fed. On the same data and hardware, a 3.35M autoregressive byte-GPT-2 reaches 2.93 bits/byte — so Physis-LM is currently about **1.44x the per-byte loss of a comparable byte-transformer baseline**. This gap is structural at this scale (see study S12 on target economics), not a tuning gap, and is not closed in this release. The value here is a faithful, thoroughly tested implementation of the architecture and a documented record of what helps and what does not.

## One-call training (0.1.4)

```python
from physis_lm.fit import fit

report = fit("my_corpus.txt", out_dir="run1", minutes=15,
             C=128, Nmax=8, stride=28,          # dense-offset supervision (S12/S13)
             ss_prob_max=0.0,                    # scheduled sampling opt-in (S14 trade-off)
             compile=False)                      # opt-in torch.compile (1.3-1.5x, note TC1)
print(report["heldout_bits_per_byte"], report["sample"])
```

`fit` handles the contamination-free split, auto batch size, the S8b memory hygiene,
checkpoint + resume, held-out logging, and ends with a diagnostics report
(`physis-lm diagnose` gives the same report from the CLI). Defaults are opinionated and
documented in the docstring; every knob maps to a measured study.

## Release quality policy (0.1.4+)

Every release must move a tracked held-out quality metric, measured and reported here
without being asked (implementers note Q1). Current ledger, on the S9 Shakespeare corpus,
16-byte held-out continuation, bits/byte (lower is better; unigram line 4.83, bigram 3.61,
same-box AR transformer 2.93):

| release | setup | bits/byte |
|---|---|---|
| 0.1.3 | one-shot Nmax=16, full recipe (S10c) | 4.89 |
| 0.1.4 | chained Nmax=4 + logits0 supervision (S13/S14/S15) | best TEACHER-FORCED **4.19** (Muon, S15); best SELF-FED **4.90** (scheduled sampling, S14) |
| 0.1.5 | all measured-positive levers stacked (S18) | new best SELF-FED **4.70**; TEACHER-FORCED **4.21**; **1.44x** the per-byte loss of byte-GPT-2 (2.93) — a structural gap NOT closed at 2.77M/CPU, stated honestly (S12/S18) |
| 0.1.6 | MoE 4-expert capacity (S19) | TEACHER-FORCED **4.21**, SELF-FED **5.16** — doubling params into experts gave NO quality gain at this scale; the bottleneck is target economics (S12), not capacity, stated honestly |
| 0.1.6 | multimodal byte ingestion (S20) | raw PNG/ZIP bytes are near-max-entropy (~8.0), unlearnable — they are already compressed |
| 0.1.6 | container DECODING fix (S21, note MM2) | `decode_containers=True` decodes first: image **pixels beat entropy by +0.40 bits** (6.30 vs 8.05 raw), gzip'd text becomes learnable (4.60) — the structure was in the decoded content, not the container |

## What's new in 0.2.0

- **JAX and TensorFlow full-model parity (scope (a)) is complete.** Both backends now run the
  entire PhysisLM forward -- PLRB, the DDHH hourglass, the bottleneck, and SConM -- matching
  the PyTorch reference to machine precision (~1e-16 on logits) at copied weights in float64,
  and greedy generation is byte-identical. Training remains in PyTorch; JAX/TF are
  forward/generation parity backends. (Notes J2/T2.)
- **A measured ~45x multimodal quality gain on the image path.** Delta-encoding decoded pixels
  (a reversible PNG-Sub-style transform) drops a byte model's loss on image pixels from ~5.1 to
  ~0.11 bits/byte, because it exposes adjacent-pixel similarity as low-entropy residuals. Still
  byte-level, not image understanding. (Note MM3, study S22.)
- **Robustness hardening:** clearer typed errors on bad input, corrupt/truncated archive
  detection, and broad edge-case coverage, so the library fails clearly instead of cryptically.
- The core text model's honest ceiling is unchanged: ~4.2 bits/byte at this scale (~1.44x a
  byte-transformer baseline), a structural gap this release does not claim to close.

## What's new in 0.1.3

- **TensorFlow parity backend** (`physis_lm.tf_backend`, optional `[tf]` extra):
  the primitive layers and a full PhysisBlock over torch-copied weights,
  parity-tested vs torch to ~1e-10 in float64 (12 tests). Same stated boundary
  as the JAX backend: the full end-to-end model is NOT ported (note TF1).
- **Teacher-model distillation** (`torch_backend/distill.py`): soft/hard
  behavioural distillation into a Physis student with an optional representation
  aligner -- explicitly NOT an "exact replica" of a teacher (14 tests).
- **NoProp-style training, switchable vs backprop** (`torch_backend/noprop.py`):
  a parallelism-preserving denoising chunk decoder trained end-to-end or
  layer-locally (no cross-block gradients; after Li, Teh & Pascanu 2025,
  adapted). Measured honestly: cheaper per step, behind backprop in accuracy at
  equal wall-clock at toy scale (study S10b).
- **Optional C++ accelerator** (`physis_lm._native`): the PCC RLE pre-pass in
  pybind11 C++, byte-identical to the Python reference, ~70x faster; pure-Python
  fallback always available (note N1).
- **Why transformers out-learn this architecture at tiny compute, measured**
  (studies S10/S10c): same data, same box -- a 3.35M AR byte-GPT-2 reaches 2.93
  bits/byte and is still descending where Physis (plain CE *and* the full
  package recipe) plateaus at ~4.8-4.9; ~34x supervision-density difference
  measured; reasons ranked, direction for parallelism-preserving fixes stated.

## What's new in 0.1.2

- **The model now demonstrably generalizes on a task that requires routing a
  transformation through the bottleneck — after fixing a bug that made it look
  like it couldn't.** `AdaptiveSlotPool` (Section 10.1) was the only attention in
  the model that did not RMSNorm its input before the K/Q projections; combined
  with the shared small init, its pre-softmax scores were std ~1e-7, softmax was
  uniform for all M slot queries, and the deep bottleneck latent Z collapsed to M
  copies of one vector (measured: cross-slot cosine 1.0000). A *copy* task hid
  this (the output->input cross-attention bypasses Z); a Caesar-*shift* task
  exposed it — training sat at chance. **Fix (bug S2):** RMSNorm before the K/V
  projections + a 1/sqrt(C) init on WQs/WKs. After it, a 3.7M and a 14.7M model
  both learn shift+3 to **100% held-out byte AND sequence accuracy** on a
  seed-disjoint test set (26^16 prompt space, so this is rule-learning, not
  memorization). See `docs/SCALE_FINDINGS.md` study S8.
- **On real text, it does NOT yet produce meaningful language — and the reason is
  compute, not the architecture.** Trained on 1.1M bytes of Shakespeare
  (next-chunk prediction, an 11M model on CPU), the model's loss falls to ~4.8
  bits/byte within ~150 steps and then plateaus: it learns the marginal character
  distribution (its samples are a single repeated character) and little context
  structure. This reproduced across three chunk lengths and two batch sizes. A
  byte model needs many epochs over even this small corpus to learn language, and
  at ~0.5-2s/step on a 4GB CPU box the model sees well under one epoch; plain
  cross-entropy (without the designed auxiliary losses) and the parallel-chunk
  objective compound it. Whether the architecture models language at real scale
  is untested here — gated by hardware, not answered. Study S9.
- **Section 3b — Hugging Face wrapper (`physis_lm.hf`).** `PhysisLMConfig` +
  `PhysisLMForByteModeling` round-trip through `save_pretrained`/`from_pretrained`
  and register with `AutoModel`. It is non-autoregressive, so `generate()` RAISES
  and points at `generate_cpd` rather than silently running HF's AR decoder.
  Optional import; `transformers` is not a hard dependency.
- **Section 3a — JAX parity backend (`physis_lm.jax_backend`).** The primitive
  layers and a full `PhysisBlock`, ported as pure functions over torch-copied
  weights and parity-tested to ~1e-10 in x64 (12 tests). Scope is stated plainly:
  the full end-to-end hourglass/PLRB/SConM forward is **not** ported (note J1).
- **Appendix D — power-iteration spectral norm (`spectral_norm_method=
  "power_iteration"`).** The fused CUDA kernels remain out of scope (no GPU), but
  the numerical method they would accelerate is implemented and selectable;
  converges to the exact SVD sigma_max. Its autograd surrogate makes gradcheck
  fail *by design* — documented, not hidden (note D1).
- **Honest scale ceiling.** On the 4GB CPU sandbox, models above ~15M are killed
  by transient out-of-memory spikes mid-training (PLRB's content-dependent tensor
  shapes fragment the allocator) even at ~1GB steady RSS; `malloc_trim` + bs=4
  made 14.7M stable, and beyond that needs more RAM or a GPU. This is an
  environment limit, not an architecture result (study S8b).

## What's new in 0.1.1

Four of the reference's documented findings now have fixes. Each was
re-reproduced against 0.1.0 before being touched, is covered by a new test, and
— except F5, which only changes an initialization and is a strict improvement —
is **off by default**, so every 0.1.0 result still reproduces exactly.

- **F5 fix (ON by default):** the attention-pool downsampler is now
  norm-preserving (W_V init std 1/sqrt(C) + a learnable gain), so the residual
  stream no longer collapses below RMSNorm's eps floor at init. Strict
  improvement; `pool_norm_preserving=False` restores the old init.
- **R2 fix (opt-in prototype):** `plrb_soft_routing=True` gives PLRB a
  differentiable soft-routing path so `W_route` actually trains — the gradient
  route the paper claims but the literal argmax doesn't provide.
- **F3 fix (opt-in prototype):** `generate_cpd(streaming=True)` now runs, via
  periodic PLRB re-hashing (Remark 15.1). Exact (bit-identical to naive CPD) at
  `rehash_period=1`. It does **not** reach the paper's O(L_out) cost — the DDHH
  still recomputes each chunk, because its coarse levels genuinely can't be
  cached (finding F3 stands) — and that is stated, not hidden.
- **F4 option (opt-in):** `fixed_depth_padding=True` pins ell/n0 to a constant
  graph. A true logit-level padding-length invariance was investigated and shown
  structurally impossible for this architecture; see "Batching and
  train/inference consistency" below.
- **Thinker-mode (opt-in prototype):** `thinker_steps > 1` (and optional
  `thinker_scratchpad_slots`) iterates the deep bottleneck over the latent as a
  parallelism-preserving deliberation loop (no autoregressive fallback). The
  mechanism is live and trainable; it is **not** a demonstrated reasoning gain —
  the extra steps must be trained to do useful work. See "Thinker-mode" below.
- License is now **Apache-2.0** (was MIT).

Full detail and honest limits: `docs/IMPLEMENTERS_NOTES.md` (notes F3, F4, F5,
R2) and `docs/SCALE_FINDINGS.md` (studies S3, S6, S7).

## Read this first (honesty statement)

**Physis-LM is currently in the demo stage; we cannot guarantee that everything
works. If you train the model and get poor results, please be aware that this
could be one of the contributing factors.**

The paper's own title page states that **no version of this architecture has
ever been assembled, trained, or evaluated end-to-end**, and that every quality
claim in its Section 20 is an untested prediction. This package changes the
first half of that sentence — the architecture is now assembled, correct at the
paper's own toy scale, and heavily tested — and none of the second half:

- **There are no pretrained weights.** Anywhere. `physis-lm train` really
  trains (and toy-scale tests verify learning happens), but training a
  competitive model needs the paper's 50-500B-byte budgets and GPUs.
- Everything implemented is tested against the paper's own numbered claims;
  everything not implemented raises a clear error or is listed below.
- Three paper claims are **demonstrably false as written** (findings F3, F4,
  F5) and one is self-contradictory (R2); 0.1.1 adds an honest fix or mitigation
  for each (see "What's new" above) while the underlying findings are still
  documented in full, not papered over. See `docs/IMPLEMENTERS_NOTES.md` and
  `docs/SCALE_FINDINGS.md`.

## Install

```bash
pip install physis-lm            # from a built wheel/PyPI
# or, from a source checkout:
pip install -e ".[test]" && pytest
```

Dependencies: `numpy`, `torch>=2.1` (CPU is fine — this package is developed
and tested entirely on one CPU core). Python >= 3.10.

## Quickstart — library

```python
import torch
from physis_lm import PhysisCoreConfig, PhysisLM

cfg = PhysisCoreConfig(C=32, M=8, H=4, W=8, Nmax=32, nref=2048,
                       NB=2, K_sconm=2, droute=8, R=2, bucket_target=8)
model = PhysisLM(cfg)

ids = torch.tensor([list(b"hello physis, this is a byte-native model")])
out = model(ids)                       # single parallel forward pass
print(out.logits.shape)                # (1, Nmax, 256) — all positions at once
print(out.sigma_score)                 # LP completion score (Section 12.6)

text, info = model.generate_cpd(ids, Lmax=128)   # Chunked Parallel Decoding
print(bytes(text.tolist()))
```

Training (the full Section 18.4 three-stage schedule, homoscedastic loss
weighting, curriculum, checkpointing):

```python
from physis_lm import data as D
from physis_lm.torch_backend import train as TR

ds = D.dataset_from_paths(["my_corpus/"], context_len=128,
                          Nmax=cfg.Nmax, pad_byte=cfg.pad_byte)
provider = D.batch_provider_from_dataset(ds, batch_size=4, pad_byte=cfg.pad_byte)
history = TR.train(model, provider, TR.TrainSettings(total_steps=1000,
                                                     ckpt_dir="ckpt"))
```

The model pads internally to exactly `M * r**ell` (the CV 15.3 hard
requirement), so **you never need to understand DDHH depth arithmetic**: any
input length up to `cfg.max_supported_raw_length() - Nmax` just works, and
anything else raises an error telling you to increase `nref`.

## Quickstart — CLI

```bash
physis-lm selftest                     # end-to-end pipeline check (~5 s)
physis-lm info --size base             # configs + parameter-count oracles
physis-lm train --data my_files/ --steps 500 --ckpt-dir ckpt
physis-lm generate --ckpt ckpt/final.pt --prompt "hello" --max-bytes 200
physis-lm scale-study                  # regenerate docs/SCALE_FINDINGS.md
```

`train` accepts `.txt`, `.jsonl` (`--text-field`), `.csv`/`.tsv`, `.html`
(`--html-mode strip|raw`), and raw binary files; malformed inputs fail with
messages naming the file, line, and problem.

## Status table

Per-module status; each module's docstring carries the same note, and
`docs/IMPLEMENTERS_NOTES.md` holds the full list of paper
ambiguities/contradictions and the decision taken for each (tags like "note B7"
below resolve there).

| Module | Paper | Status | Tested by | Limitations / notes |
|---|---|---|---|---|
| `core` geometry, padding, output slice | 7.2, Rem. 7.1, CV 15.3, Rem. 15.4 | implemented | test_core (incl. the literal CV 15.3 sweep, hypothesis sweeps) | ell >= 1 clamp (G1) |
| `core` complexity + parameter oracles | 7.7, C.1/C.6, Prop 9.1, 17.5, Sec. 1, 12.2, 18.7, App. A | implemented | test_core reproduces every printed table/figure | paper's 130,808 misprint recorded (P1) |
| `core` LSH oracle + config | 9.3, Prop 9.2, CV 9.3, App. A.1 | implemented | test_core (collision/recall sims; config rejections) | Omega = f(seed, round, B) (R1) |
| RMSNorm / RoPE / SwiGLU / spectral / local attention | 6.2-6.6, 28.1-28.4, CV 8.3, Prop 8.2 | implemented | test_torch_layers (banded==O(n^2) oracle, CV 8.3 both halves, circulant Jacobian, gradchecks) | banded impl is O(nW) memory; no CUDA kernels (App. D out of scope, no GPU) |
| PhysisBlock, pooling, upsample, bottleneck, LP | 6.5, 7.3-7.4, 10, 12, CV 8.5, CV 12.6 | implemented | test_torch_blocks (CV 8.5 both cases, all three LP propositions vs autograd) | B3, B8-B10 readings; 0.1.2 fixes AdaptiveSlotPool bottleneck-collapse bug S2 (RMSNorm + 1/sqrt(C) init) |
| PLRB | 8-9, 19.5, C.6 | implemented (fixed hashing; 0.1.1 opt-in soft routing) | test_torch_plrb (segmented == mask oracle, sentinel, zero W_route grad by default, soft-routing gives W_route a real gradient) | default trains everything except W_route as literally specified (R2); 0.1.1 `plrb_soft_routing` fixes this (prototype); learnable-Omega variant a disclosed stub (R3); dense hashing = C.6 trap at large n (study S5) |
| SConM + streaming readout | 14, CV 14.2, Lemma 14.8, D.2 | implemented | test_torch_sconm (LOO exact single/multi-head, LSE recovery, vertex bound, scopes) | no contraction claim — conditional on unmeasured L_g (S1; studies S1/S2); 0.1.2 adds opt-in power-iteration spectral norm (note D1) |
| Full model, integration, LSR, CPD | App. B, CV 15.2/15.3, Rem. 15.4, 7.8.2, 15 | implemented | test_torch_model (the paper's own sixteen-join toy-scale integration test, ablations, LSR bit-exactness, CPD incl. streaming exact-at-period-1) | 0.1.1 streaming CPD runs via periodic PLRB re-hash, exact at rehash_period=1, but does not reach O(L_out) — DDHH still recomputes (finding F3); CPD stops gracefully at the context limit |
| Losses, schedules, training loop, PTCC | 16.3, 18, 19, CV 18.2/18.4/18.10 | implemented | test_training (a real toy run that learns + checkpoint round-trip), test_data_ptcc_pcc | L1, T1-T3 decisions; no mixed precision / Flash Attention / grad checkpointing (throughput devices, no GPU here) |
| PCC | 24, CV 24.2 | implemented | test_data_ptcc_pcc (losslessness incl. adversarial streams, tier accounting) | ratios are corpus claims — only CV 24.2's qualitative pattern asserted (C1) |
| Data layer + CLI | task Sec. 5 | implemented | test_data_ptcc_pcc, test_cli (end-to-end train->generate on real files) | formats defined precisely in data.py's docstring |
| Scale studies | task Sec. 2 | implemented | test_scale_studies | untrained instantiations only; see report header |
| Thinker-mode (deliberation loop) | task Sec. 5 | prototype (opt-in) | test_torch_model (default == single-pass, live + trainable, parallelism preserved) | mechanism only; genuine reasoning gains require training the steps (note K1) |
| JAX parity backend (layers + PhysisBlock) | task Sec. 3a | implemented (layer/block level) | test_jax_parity (12 tests, torch-vs-JAX at copied weights, ~1e-10 in x64) | full end-to-end hourglass/PLRB/SConM forward NOT ported — tested foundation only (note J1) |
| Hugging Face wrapper | task Sec. 3b | implemented | test_hf (config round-trip, save/from_pretrained identical logits, AutoModel, non-AR contract) | optional import; `generate()` raises -> `generate_cpd`; no pretrained weights |
| Appendix D spectral-norm kernel | App. D | power-iteration path implemented; fused CUDA kernels out of scope | test_torch_sconm (converges to SVD sigma_max; gradient flow) | no GPU for fused kernels; gradcheck fails by design (surrogate gradient, note D1) |
| Any-to-Physis weight transplant | task Sec. 4 | **deferred to a later release** | — | intentionally not shipped in this version; design in docs/SECTIONS_3_4_DESIGN.md |
| NoProp-style training (switchable vs backprop) | 0.1.3 dev | prototype (unreleased) | test_noprop_tf (gradient locality, oracle chain, both methods learn, registry) | layer-local trainer on its own parallel denoising decoder, NOT a PhysisLM flag; measured behind backprop at equal wall-clock (study S10b, note NP1) |
| Exact parameter counts (`param_budget`) | 0.1.4 | implemented | test_014_features (exact-count asserts, minimal filler, inert calibration) | non-functional calibration residual documented (note P1) |
| LoRA + QLoRA-int8 base (`torch_backend/lora`) | 0.1.4 | implemented | test_014_features (identity at attach, merge, adapter round-trip, int8 memory-freed + error bound) | NF4/bitsandbytes out of scope on CPU (note LR1) |
| Diagnostics (`diagnostics`, `physis-lm diagnose`) | 0.1.4 | implemented | test_014_features (reference entropies, per-position CE, head report) | the S11 audit, productized (note DG1) |
| Dense-offset supervision (`stride`) + decode heads | 0.1.4 | implemented | test_014_features | quality gate S13: 4.89 -> 4.26 teacher-forced (self-fed regression stated) |
| Mixture-of-Experts FFN (`use_moe`, top-k, load balancing) | 0.1.6 | implemented | test_016_moe (dense parity 1e-12, routing, load balance, capacity, aux training) | opt-in; adds capacity at ~const per-token FLOPs; quality in S19 |
| Multi-device: CPU threads / GPU DataParallel+DDP / TPU-NPU-XPU resolution | 0.1.6 | implemented | test_016_devices (CPU paths + absent-hardware errors) | **GPU/TPU/NPU provided but NOT executed on hardware here** (note DEV1) |
| Native C++ LSH kernel (bit-identical to numpy) | 0.1.6 | implemented | test_016_native_lsh (200 shapes, ties, contiguity) | ~0.8-1.0x numpy — completeness, not a speedup (note CPP1) |
| Torch-native LSH bucketing (exact parity; numpy reference retained) | 0.1.5 | implemented | test_015_speed (exact ids, ties, sentinel, bit-identical logits) | 1.01x measured -- cleanup, not a speed claim (note SP1) |
| Learnable-soft routing (`plrb_soft_routing`) | 0.1.5 | measured (S17) | S16 census + S17 study | 1.59x training speed at TF parity; quality bar not met, stated (S17) |
| In-process error handling: OOM microbatch recovery, non-finite rollback (guard) | 0.1.5 | implemented | test_015_resilience (unit + train integration) | distinct from checkpointing; SIGKILL not catchable, stated (note GD1) |
| Built-in autosave + signal emergency capture + save_pretrained | 0.1.5 | implemented | test_015_resilience (rotation, atomic, verify, best, real-SIGTERM subprocess) | note AV1 |
| Seamless resume (bit-for-bit, RNG-complete) | 0.1.5 | implemented | test_015_resilience (exact loss-sequence equality) | closes 0.1.4 resume caveat (note SR1) |
| Decode sampling / repetition penalty / entropy-commit | 0.1.4 | implemented | test_014_cycle2 (validation, determinism, commit bounds, backcompat) | adaptive commit measured NEGATIVE on non-SS models (note AC1) |
| Chunk-level scheduled sampling | 0.1.4 | implemented | test_014_cycle2 (ramp, eligibility, off=identical) | study S14 (note SS1) |
| Muon optimizer + hybrid builder (`optimizer="muon"`) | 0.1.4 | implemented | test_014_cycle2 (NS spectrum, split, convergence, state, TR selection) | S15: TF 4.19 vs 4.26 at equal wall-clock, self-fed slightly worse; opt-in (note MU1) |
| QLoRA NF4 base (bitsandbytes CPU, probed) | 0.1.4 | implemented (probe-gated) | test_014_cycle2 (skips where unavailable) | Linear4bit gemm NOT used/needed; persistence limits stated (note BN1) |
| fit() one-call training (+ opt-in torch.compile) | 0.1.4 | implemented | test_014_cycle2 (train/resume/report) | compile measured 1.33-1.51x (notes FT1/TC1) |
| muP-style width scaling | 0.1.4 | **experimental** | test_014_cycle2 (init/lr rules, coordinate check) | HP transfer unvalidated (note MP1) |
| TensorFlow parity backend (layers + PhysisBlock) | 0.1.3 | implemented (layer/block level) | test_tf_parity (12 tests, torch-vs-TF at copied weights, ~1e-10 in float64) | full end-to-end hourglass/PLRB/SConM forward NOT ported, same boundary as JAX (note TF1); optional `[tf]` extra |

Test suite: **10,254 tests, all passing** (heavily property-based) (`pytest -q`), dominated by cheap
parametrized pure-math sweeps plus hypothesis-generated cases; every
Computational Verification remark in the paper that can run on CPU is
re-executed against this codebase's actual code.

## The findings and their 0.1.1 fixes

- **F3** — Streaming DDHH's premise ("levels above j* do not change") is false:
  spectral mixing is a global circular convolution at every level (quantified in
  study S7). *0.1.1:* `generate_cpd(streaming=True)` now runs — bit-identical to
  naive CPD at `rehash_period=1`, and periodic PLRB re-hashing (Remark 15.1) for
  larger periods — but the DDHH still recomputes each chunk, so it does not reach
  the paper's O(L_out) streaming cost. The finding stands; the fix is honest
  about what it does and doesn't buy.
- **F4** — the literal Section 19.5 claim that padding to a longer batch length
  leaves behavior identical is structurally impossible (padding changes ell, and
  extending the input shifts the output placeholders). The achievable invariant
  — batch-content independence at fixed shape — holds to 1e-10 and is tested
  (study S6). *0.1.1:* `fixed_depth_padding=True` pins ell/n0 to a constant
  graph; a true logit-level length invariance was investigated and shown
  impossible here.
- **F5** (found by this implementation) — at initialization the residual stream
  collapsed geometrically through the attention-pool downsamplers (~x0.02-0.09
  per level), pushing deep levels under RMSNorm's eps floor (study S3). *0.1.1
  (ON by default):* the norm-preserving downsampler holds every level above the
  floor across the whole grid measured; S3 is now a before/after table.
- **R2** — Section 9.7 is self-contradictory: it says the routing argmax "is not
  differentiated through" yet "gradients flow normally through W_route". The
  literal argmax gives W_route exactly zero gradient (tested), so the default
  fixed-hashing path trains everything except W_route. *0.1.1:*
  `plrb_soft_routing=True` supplies a differentiable soft-routing path so
  W_route trains (prototype; off by default).

## Batching and train/inference consistency

Because n0 = M * r**ell depends on the padded input length, this architecture's
output for a fixed sequence is **not** invariant to how much padding you add:
different padded lengths mean a different number of DDHH levels and a different
spectral DFT length (finding F4). Two practical consequences:

1. The guarantee you *can* rely on is **batch-content independence at a fixed
   batch shape**: a sequence's logits do not depend on which other sequences
   share its batch, as long as the batch's padded length is the same. This holds
   to floating-point precision and is tested.
2. To keep training and inference consistent, either (a) use the **same nref and
   the same batch-shaping discipline** in both — so a given input lands at the
   same ell in training and at inference — or (b) set
   `PhysisCoreConfig(fixed_depth_padding=True)`, which pins every input to
   ell_max / n0 = M*r**ell_max. Option (b) makes the computation graph constant
   across input lengths (removing the batch-shape-dependent DFT-length drift) at
   the cost of always running at maximum depth; it does not make two
   different-length inputs produce identical logits (that is impossible here),
   but it removes padding-length as a source of train/inference mismatch.

## Thinker-mode (prototype)

`PhysisCoreConfig(thinker_steps=T, thinker_scratchpad_slots=S)` turns on an
opt-in deliberation loop (task Section 5). Instead of a single deep-bottleneck
pass over the compressed latent Z, the bottleneck is applied T times, adding a
learnable per-step code each iteration, with S optional learnable "scratchpad"
latent slots as working memory (dropped before readout so the output shape is
unchanged). Every latent slot is refined simultaneously at each step, and the
loop iterates over deliberation steps, never over output tokens — so the
architecture's parallelism is preserved and there is **no** autoregressive
fallback. `thinker_steps=1` with no scratchpad (the default) is byte-for-byte
the single-pass model.

What this is and isn't: the mechanism is implemented, live (non-zero step codes
change the output), and trainable (gradients reach the step codes and
scratchpad) — all tested. It is **not** a demonstrated reasoning improvement.
The per-step codes are zero at initialization, so an untrained model's first
step equals the ordinary pass and further steps just re-apply the bottleneck;
making the extra steps do genuinely new work is a *training* problem (train with
`thinker_steps > 1`, most plausibly with a ponder/ACT-style objective). This
release ships the mechanism honestly labeled as a prototype, not a claim that
iterating an untrained bottleneck reasons better. See `docs/IMPLEMENTERS_NOTES.md`
note K1.

## Repository layout

```
physis_lm/
  core.py            geometry, cost/parameter oracles, LSH math, config (no torch)
  data.py            byte-level corpus loading + batching (stage-1 padding only)
  pcc.py             Physis Context Compression (Section 24; no torch)
  cli.py             the `physis-lm` command
  scale_studies.py   Section-2 measurements -> markdown report
  torch_backend/
    layers.py blocks.py plrb.py sconm.py model.py losses.py train.py ptcc.py
tests/               the full suite; conftest.py holds the paper's toy scale
docs/                PLAN.md (pre-implementation plan), IMPLEMENTERS_NOTES.md,
                     SCALE_FINDINGS.md, SECTIONS_3_4_DESIGN.md
```

## Citing

If you use this implementation, cite the Physis-LM preprint (Omur Bera Isik,
2026). This codebase: `physis-lm` 0.2.4, made with AI assistance, Apache-2.0 license.
