Metadata-Version: 2.5
Name: pantogloss
Version: 0.12.0
Summary: TensorFlow/Keras many-to-English machine translation
Project-URL: Homepage, https://github.com/chrismattmann/pantogloss
Project-URL: Repository, https://github.com/chrismattmann/pantogloss
Project-URL: Model repository, https://huggingface.co/chrismattmann/pantogloss-500-en
Project-URL: FP16 model repository, https://huggingface.co/chrismattmann/pantogloss-500-en-fp16
Project-URL: Issues, https://github.com/chrismattmann/pantogloss/issues
Author: Chris A. Mattmann
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: keras,machine translation,multilingual,tensorflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub<2,>=0.26
Requires-Dist: nlcodec<0.6,>=0.5
Requires-Dist: numpy<3,>=1.26
Requires-Dist: sacremoses<0.3,>=0.2
Requires-Dist: tensorflow<2.19,>=2.18
Provides-Extra: conversion
Requires-Dist: ruamel-yaml>=0.17; extra == 'conversion'
Requires-Dist: torch<3,>=2.2; extra == 'conversion'
Provides-Extra: cuda
Requires-Dist: tensorflow[and-cuda]<2.19,>=2.18; extra == 'cuda'
Provides-Extra: evaluation
Requires-Dist: sacrebleu<3,>=2.4; extra == 'evaluation'
Provides-Extra: metal
Requires-Dist: tensorflow-metal<1.3,>=1.2; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'metal'
Requires-Dist: tensorflow<2.19,>=2.18; extra == 'metal'
Provides-Extra: test
Requires-Dist: pypdf<7,>=6.16; extra == 'test'
Requires-Dist: pytest-cov<8,>=5; extra == 'test'
Requires-Dist: pytest<10,>=8; extra == 'test'
Requires-Dist: sacrebleu<3,>=2.4; extra == 'test'
Provides-Extra: tika
Requires-Dist: pypdf<7,>=6.16; extra == 'tika'
Requires-Dist: tika<4,>=3.3; extra == 'tika'
Description-Content-Type: text/markdown

# Pantogloss

[![Tests](https://github.com/chrismattmann/pantogloss/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/chrismattmann/pantogloss/actions/workflows/test.yml)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](LICENSE)

Pantogloss is a TensorFlow/Keras many-to-English machine-translation library.
Its first model, `pantogloss-500-en`, was converted and numerically validated
from the model described in *Many-to-English Machine Translation
Tools, Data, and Pretrained Models* (ACL-IJCNLP 2021).

The Python package is distributed through PyPI, while its models are kept in
separate public Hugging Face repositories. Pantogloss downloads them
anonymously by default; cached or explicit Hugging Face credentials remain
supported for private and gated model repositories.

The codebase and converted models are licensed under Apache-2.0.

## Intended API

```python
from pantogloss import Translator

translator = Translator.from_pretrained("pantogloss-500-en")
print(translator.translate("Comment allez-vous ?"))
```

Pantogloss 0.11 and later also provide the validated FP16 model. It is about
50% smaller and substantially reduces accelerator/process memory while keeping
the original float32 model as the default:

```python
translator = Translator.from_pretrained("pantogloss-500-en-fp16", device="gpu")
```

The complete FP16 CUDA, Apple Metal, and 8,250-example quality evidence is
documented in [the FP16 experiment](docs/fp16-experiment.md).

Discover released models without importing TensorFlow:

```python
from pantogloss import available_models

for model in available_models():
    print(model.name, model.precision, model.repository, model.revision)
```

Greedy decoding remains the default because the full evaluation found it about
16 times faster than beam-4. The validated quality preset enables beam-4 with
length penalty 0.6 without changing the return type:

```python
print(translator.translate("Comment allez-vous ?", preset="quality"))
```

Use `preset="fast"` for the explicit greedy preset, or `preset="custom"` with
`beam_size` and `length_penalty` for advanced tuning. Explicit decoding values
also override either standard preset.

The CLI exposes the same choice:

```bash
pantogloss translate --preset fast "Comment allez-vous ?"
pantogloss translate --preset quality "Comment allez-vous ?"
pantogloss translate --preset custom --beam-size 8 --length-penalty 1.0 \
  "Comment allez-vous ?"
```

Pantogloss selects the first TensorFlow GPU automatically and enables memory
growth. Device choice can also be made explicit:

```python
translator = Translator.from_pretrained("pantogloss-500-en", device="gpu")
print(translator.device_info)
```

Using `device="gpu"` fails clearly if TensorFlow cannot see a GPU; use
`device="cpu"` to force CPU inference.

Greedy and beam translation use encode-once, graph-compiled TensorFlow decoding
loops with decoder self-attention and cross-attention key/value caches by
default. If a TensorFlow backend cannot compile a loop, Pantogloss falls back
to its eager cached decoder. Eager execution can also be selected explicitly:

```python
translator = Translator.from_pretrained("pantogloss-500-en", compiled_decode=False)
```

For exact comparison with the original full-prefix beam calculation, construct
the translator with `cached_beam=False`. This slower numerical reference may
choose different near-tied beam candidates than incremental cached attention.

Structured results are additive to the original string API:

```python
result = translator.translate_detailed("Bonjour le monde.")
print(result.text, result.source_tokens, result.target_tokens)
print(result.execution_device, result.elapsed_seconds)
```

Caller-supplied language hints can attach versioned benchmark evidence and
opt-in, model-independent warnings to structured results:

```python
result = translator.translate_detailed(
    "ສະບາຍດີ",
    source_language="lo",
    include_warnings=True,
)
print(result.language_quality.tier, result.language_quality.measured_chrf)
for warning in result.warnings:
    print(warning.kind, warning.message)
```

Pantogloss does not detect the source language. `source_language` accepts an
ISO 639-1 or BCP-47 hint from the caller; a batch may provide one hint or an
aligned sequence. Tiers are aggregate evidence from one pinned FLORES+ corpus,
not per-sentence confidence or a guarantee. Unmeasured languages are labeled
`unmeasured`, never unsupported. `measured-higher` is chrF 60 or above,
`measured-mixed` is 45 through 59.99, and `measured-limited` is below 45.
Warnings are disabled by default and can flag measured-limited or unmeasured
languages, exact source copies, retained source scripts, repetition, empty
output, and extreme length ratios.

For extracted documents, one shared model can segment, batch, and reconstruct
text while retaining blank lines and paragraph boundaries:

```python
from pantogloss import DocumentTranslator

documents = DocumentTranslator(translator)
result = documents.translate(
    extracted_text,
    source_language="fr",
    max_source_tokens=512,
    long_input="split",
)
print(result.text)
```

Long segments can use `split`, `truncate`, or `error` policy. Failures are
isolated to individual segments and recorded in `result.segments`; successful
neighbors remain ordered. Layout-only segments such as page numbers, dot
leaders, and separator rules are preserved verbatim instead of being sent to
the translation model.

Install `pantogloss[tika]` to use the `pantogloss-tika` extraction-to-English
command without a translation server or Docker container. It supports bounded
trials and page ranges, displays progress, writes output incrementally, and
resumes from a fingerprint-protected JSONL checkpoint. See
`examples/tika_to_english.py` for its compatibility wrapper.

The same workflow is available as a stable Python API:

```python
from pantogloss.tika import translate_document

result = translate_document(
    "report.pdf",
    output="report.en.txt",
    device="auto",
)
print(result.diagnostics)
print(result.runtime)
```

Apple Silicon GPU document runs automatically use short-lived TensorFlow
workers, committing five chunks before each worker exits. This bounds Metal's
retained unified-memory allocations without changing translations or checkpoint
compatibility. Tune the interval with `metal_worker_chunks=N` in Python or
`--metal-worker-chunks N` on the command line; use `None` in Python or `0` on
the command line to disable recycling. CPU and CUDA runs remain in-process.

New checkpoints record per-chunk diagnostics, translation timing, throughput,
peak process memory, and effective device metadata. Inspect or validate them
offline without importing TensorFlow or Tika:

```bash
pantogloss-tika inspect report.en.txt.jsonl
pantogloss-tika validate report.en.txt.jsonl --output report.en.txt
pantogloss-tika review report.en.txt.jsonl --output review.jsonl
pantogloss-tika review report.en.txt.jsonl --format csv --output review.csv
pantogloss-tika review report.en.txt.jsonl \
  --include-preserved formula --format csv --output formulas.csv
```

Diagnostics are conservative review signals—not translation-quality scores.
They flag empty output, retained multi-character source-script runs, fourfold
word repetition, and extreme character-length ratios; scientific symbols such
as `α`, `β`, and `π` are intentionally not treated as untranslated prose.
The model-free `review` command joins every finding back to its aligned source
and translation with chunk, segment, character-offset, token, truncation, and
error metadata. JSONL is the default for automated processing; CSV is convenient
for spreadsheet review.
Use `--include-preserved formula`, `layout_only`, or `all` to add informational
rows for deliberately untranslated spans without turning them into diagnostic
warnings. During classifier development, `tools/audit_formula_detection.py`
replays the explainable formula policy over an existing checkpoint and can emit
candidate-level JSONL with signal counts, natural-word count, and math density.

Formula-heavy spans are preserved verbatim by default because general-purpose
translation models can turn extracted equations into plausible but invented
prose. Checkpoints record these spans with `preservation_reason="formula"`, and
`inspect` reports preservation counts. Normal prose containing occasional
mathematical notation remains translatable. Whole-span preservation requires
zero natural-language words; mixed prose and notation stays in the translation
and diagnostic path rather than silently retaining source-language prose. Use
`preserve_formulas=False` with `DocumentTranslator.translate()` or
`translate_document()`, or pass
`--translate-formulas` to `pantogloss-tika`, to restore the prior behavior. The
Tika choice is protected by the checkpoint fingerprint.

Schema-1 checkpoints created by Pantogloss 0.5 remain inspectable and can be
resumed when their source checksum, model revision, and decoding options match.

Install the accelerator backend for the machine:

```bash
# Linux with an NVIDIA GPU
pip install 'pantogloss[cuda]'

# Apple Silicon
pip install 'pantogloss[metal]'
```

Both use the same `device="auto"` or `device="gpu"` Python API. The CUDA extra
does not install or replace the host NVIDIA driver. The Metal extra uses Apple's
TensorFlow PluggableDevice and the TensorFlow 2.18 runtime combination validated
by the Bytewise project.

Source batches use bounded power-of-two padded widths by default (for example,
16, 32, and 64 through the active source-token limit). Padding remains masked
and does not change source token counts. This bounds accelerator allocation
shapes for long document runs; use `source_padding="exact"` with
`Translator.from_pretrained()` or `--source-padding exact` as a reference mode.
The policy and actual padded width are included in runtime and segment metadata
and the policy is protected by Tika checkpoint fingerprints.

Greedy decoding runs on the selected device. On Apple Silicon, beam decoding
uses a correctness-first CPU execution fallback because Panto-500 validation
found shape-sensitive corruption in Metal beam-expanded inference. CUDA beam
decoding remains on GPU. Pantogloss records the effective beam execution device
in evaluation manifests instead of silently claiming Metal placement.

The models are stored separately in the public Hugging Face repositories
[`chrismattmann/pantogloss-500-en`](https://huggingface.co/chrismattmann/pantogloss-500-en)
and
[`chrismattmann/pantogloss-500-en-fp16`](https://huggingface.co/chrismattmann/pantogloss-500-en-fp16);
neither is included in the Python wheel.

The default `token=None` uses a locally cached Hugging Face credential when one
exists but does not require one for public repositories. Use `token=False` to
force anonymous access or pass a token explicitly without storing it:

```python
import os

translator = Translator.from_pretrained(token=os.environ["HF_TOKEN"])
```

## Command line

The `pantogloss` command loads the model once and supports arguments, files, and
line-oriented Unix pipelines:

```bash
pantogloss info
pantogloss models
pantogloss info --model pantogloss-500-en-fp16 --device gpu --json
pantogloss translate "Comment allez-vous ?"
printf 'Hola señor\nWie geht es Ihnen?\n' | pantogloss translate --device gpu
pantogloss translate --input source.txt --output english.txt --batch-size 16
pantogloss translate --beam-size 4 --length-penalty 0.6 "Hola señor"
pantogloss translate --max-source-tokens 512 --source-length-policy truncate "..."
pantogloss translate --input source.txt --preserve-empty-lines \
  --report timing.json --output english.txt
pantogloss translate --json --source-language fr --include-warnings \
  "Comment allez-vous ?"
```

Model discovery and cache inspection are also available as stable JSON. Cache
operations shown here are read-only: they neither download nor delete files.

```bash
pantogloss models --json
pantogloss models --cached --json
pantogloss cache info --json
pantogloss cache info --model pantogloss-500-en-fp16 --json
pantogloss cache verify --model pantogloss-500-en-fp16 --json
```

Use `--cache-dir PATH` with `models --cached`, `cache info`, or `cache verify`
when Pantogloss snapshots live outside the default Hugging Face cache. Verification
requires a complete released snapshot and checks every artifact size and
SHA-256 declared by its manifest.

Use `--json` for ordered JSON Lines records containing the source index,
translation, error, inference time, and preservation status. Empty input lines
are omitted by default; `--preserve-empty-lines` copies them without model
inference. Batch failures are recursively isolated so successful neighbors are
retained and the command exits 1 after writing all results; `--fail-fast`
restores immediate termination. `--input-encoding`, `--output-encoding`, and
`--encoding-errors` control file decoding and encoding. Progress is automatic
on terminals and can be forced or disabled with `--progress` or `--no-progress`.
When `--source-language` is supplied, JSONL also includes the normalized hint
and pinned benchmark evidence. Add `--include-warnings` for conservative output
review signals. These flags never add prose to plain translation output.

Use `--offline` to require an already cached model snapshot. Translation data
goes to stdout (or `--output`); model and device diagnostics are suppressed by
default so pipelines remain clean. Use `--verbose` for Pantogloss loading
progress, `--report` for a JSON timing/throughput summary, or
`--tensorflow-logs` for TensorFlow, CUDA, and Metal startup diagnostics.

## Development status

The complete 307-variable Keras model has been converted locally from all 308
learned PyTorch tensors (the target embedding and output projection are tied).
Greedy parity against the archived RTG implementation passes across a ten-language
batch: token IDs and translations match exactly, while final logits have a
maximum absolute error of 1.24e-5. With the original beam size 4 and length
penalty 0.6, all decoded four-best candidate sets match. One near-tied example
changes top rank because of framework floating-point ordering. Model version
0.1.0 is released in the public Hugging Face repository at an immutable commit.

The source model and generated artifacts stay under the ignored `artifacts/`
directory. To reproduce conversion after acquiring the source archive:

```bash
python tools/convert_rtg_checkpoint.py \
  artifacts/source/rtg500eng-tfm9L6L768d-bsz720k-stp200k-ens05 \
  artifacts/converted/pantogloss-500-en-candidate
```

Run the reference parity harness with:

```bash
CUDA_VISIBLE_DEVICES=-1 python tools/check_parity.py \
  artifacts/source/rtg500eng-tfm9L6L768d-bsz720k-stp200k-ens05 \
  artifacts/converted/pantogloss-500-en-candidate
```

To require and verify real GPU placement:

```bash
python tools/check_gpu.py artifacts/converted/pantogloss-500-en-candidate
```

## Platform validation

CUDA, CPU, and Apple Metal validation commands, benchmark methodology, release
results, memory measurements, and known backend caveats are maintained on the
[Platform Validation wiki page](https://github.com/chrismattmann/pantogloss/wiki/Platform-Validation).
The measured-language table, tier definitions, and interpretation guidance live
on the [Language Quality Catalog wiki page](https://github.com/chrismattmann/pantogloss/wiki/Language-Quality-Catalog).
This keeps changing hardware evidence out of the package overview while
preserving one reproducible validation record. Contributors preparing a release
should follow [the release checklist](docs/releasing.md).

## Project boundary

Pantogloss translates text and ordered text segments. It does not identify
languages, detect file types, or parse document formats. The existing
`DocumentTranslator` is a neutral text boundary; `pantogloss-tika` remains a
frozen compatibility example rather than a direction for new core dependencies.
See [the architecture boundary](docs/architecture.md) for how external Tika,
Bytewise, language-identification, and future neural parsing projects should
compose with Pantogloss.

## Translation evaluation

Pantogloss includes a versioned evaluation runner and a checksum-pinned,
project-authored CC0 smoke corpus covering 12 languages and seven scripts. It
supports durable resumable translation artifacts, model-free rescoring,
adaptive batch recovery, deterministic paired-bootstrap confidence intervals,
and aligned regression comparisons. Reports include BLEU, chrF, per-language
diagnostics, failures, empty and unknown-token outputs, latency, and throughput.

Install the development extras and run the complete automated suite:

```bash
# Linux CUDA
python -m pip install -e '.[cuda,evaluation,test]'

# Apple Silicon Metal
python -m pip install -e '.[metal,evaluation,test]'

python -m pytest
```

The targeted weak-language fixture adds two CC0 regression examples each for
Lao, Yoruba, Hausa, Igbo, Khmer, and Burmese, plus a public-safe per-language
diagnostic exporter. See `evaluation/README.md` for commands, checked-in
reports, reproducibility details, and the important limits on interpreting this
deliberately small regression fixture.

Detailed translation-quality methodology and aggregate results live in
`evaluation/README.md`; platform-specific execution results live only on the
[Platform Validation wiki page](https://github.com/chrismattmann/pantogloss/wiki/Platform-Validation).
