Metadata-Version: 2.4
Name: owistscribe
Version: 2.1.1
Summary: Fast, low-memory, offline OCR — CLI, local web app, and Python API in one pip install. Reads images, PDFs, and DOCX; reconstructs reading order and tables as Markdown.
Author: Openwist
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/openwist/owistscribe
Project-URL: Repository, https://github.com/openwist/owistscribe
Project-URL: Changelog, https://github.com/openwist/owistscribe/blob/main/CHANGES.md
Project-URL: Issues, https://github.com/openwist/owistscribe/issues
Keywords: ocr,text-extraction,pdf,docx,markdown,onnxruntime,document-processing,text-recognition,offline-ocr
Platform: Any
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Graphics :: Capture :: Scanners
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Text Processing
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: <4,>=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: pymupdf>=1.24.0
Requires-Dist: python-docx>=1.1.0
Requires-Dist: fastapi>=0.110.0
Requires-Dist: uvicorn[standard]>=0.29.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: colorama>=0.4.6
Requires-Dist: onnxruntime>=1.18.0
Requires-Dist: pyclipper>=1.2.0
Requires-Dist: opencv_python>=4.5.1.48
Requires-Dist: numpy<3.0.0,>=1.19.5
Requires-Dist: six>=1.15.0
Requires-Dist: Shapely!=2.0.4,>=1.7.1
Requires-Dist: PyYAML
Requires-Dist: Pillow
Requires-Dist: tqdm
Requires-Dist: omegaconf!=2.2.1
Requires-Dist: requests
Requires-Dist: colorlog
Provides-Extra: desktop
Requires-Dist: tkinterdnd2; extra == "desktop"
Provides-Extra: gpu
Requires-Dist: onnxruntime-gpu; extra == "gpu"
Provides-Extra: directml
Requires-Dist: onnxruntime-directml; extra == "directml"
Dynamic: license-file

# OwistScribe

**Openwist AI — building the future, intelligently.**

Fast, low-memory, **offline** OCR and document reading — one `pip install owistscribe` gives you
a CLI, a local web app, and a Python API, all sharing one engine. It runs on ONNX Runtime with
bundled PP-OCRv6 "small" weights — constructing the engine makes zero network calls, ever,
because the model weights ship inside the package itself. Third-party credits are in
[Credits](#credits).

```bash
pip install owistscribe
owistscribe invoice.pdf
```

## Why

- **Actually offline.** No API key, no per-page cost, no network call at inference time. The
  `.onnx` model weights are bundled in the wheel.
- **Reads more than images.** PDFs and DOCX files with a real embedded text layer are read
  directly — faster and more accurate than rasterizing and OCR'ing a picture of the text. OCR
  only runs on pages that are genuinely scanned with no text layer.
- **Output is Markdown, not a text blob.** Headings and tables are reconstructed as real
  Markdown (`#` headings, `| table | cells |`) from the OCR geometry — see
  [How extraction works](#how-extraction-works) below for what that does and doesn't cover.
- **Streams.** A multi-page PDF streams page-by-page; a single image streams region-by-region
  as text is recognized, instead of a silent wait on a large file.
- **One engine, three front ends.** The CLI, the web app, and the Python API all call the same
  `owistscribe.ScribeEngine` — a fix or a tuning change benefits all three at once.

## Install

```bash
pip install owistscribe
```

Requires Python 3.9+ (developed and tested on 3.12 and 3.13; CI runs 3.9 to 3.13 on Linux and Windows). That's it — no second package, no sibling checkout, no post-install
download step. (The drag-and-drop desktop app needs one more dependency: `pip install
"owistscribe[desktop]"`.)

## Quickstart

### CLI

```bash
owistscribe invoice.png                    # print the extracted text (Markdown)
owistscribe report.pdf --json              # structured JSON instead
owistscribe scan.jpg --min-confidence 0.8  # drop low-confidence text
owistscribe web                            # launch the web app at http://127.0.0.1:8000
owistscribe desktop                        # launch the drag-and-drop desktop app
owistscribe --help                         # every command, with examples
```

Colored terminal output (headings/tables lightly highlighted) is on automatically for an
interactive terminal, off automatically when piped or redirected, and always off with
`--no-color` or the `NO_COLOR` environment variable.

### Web app

```bash
owistscribe web
```

Opens `http://127.0.0.1:8000` in your browser. Drag in an image, PDF, or DOCX; results stream
in as they're ready and render as formatted Markdown (headings, tables) with a toggle to see
the raw text. `owistscribe web --port 9000` to use a different port.

### Python API

The shape meant for pairing with a chatbot, agent, or any app that needs to read an uploaded
file before responding:

```python
from owistscribe import Scribe

owistscribe = Scribe()                      # loads the model once — build this at startup, reuse it
result = owistscribe.read("invoice.pdf")

result.text        # the full extracted text (reading-order corrected, Markdown-formatted)
result.pages        # per-page breakdown: [PageResult(page_number=1, text=..., source=...), ...]
result.headers       # detected repeated header lines, if any
result.footers       # detected repeated footer lines, if any
```

Progressive results, for a large file or a chat UI that wants to show progress:

```python
for event in owistscribe.stream("report.pdf"):
    if event["type"] == "page":
        print(f"page {event['page_number']} ready: {len(event['text'])} chars")
    elif event["type"] == "done":
        print(f"finished in {event['took_seconds']}s")
```

One-shot convenience, if you don't want to manage a `Scribe` instance yourself:

```python
from owistscribe import read
text = read("invoice.pdf").text
```

## Supported files

Images (`.png` `.jpg` `.jpeg` `.bmp` `.tiff` `.webp`), `.pdf`, `.docx`.

## How extraction works

**Reading order & tables.** The detection/recognition models find and read individual
text boxes — they have no concept of "this is a table" or "read top-to-bottom." That's handled
separately, in `layout.py`, using plain geometry on the returned box coordinates: cluster
regions into rows by vertical-center overlap, segment by cell count so one table's structure
can't bleed into an unrelated table on the same page, detect a shared column structure, and
render it as a Markdown table when one is found — plain reading order otherwise.

**Known, tested limitation**: a genuine two-column *page layout* (a newsletter, a two-column
résumé) can be geometrically indistinguishable from an actual two-column *table* — confirmed
with an adversarial test where two unrelated columns of ordinary prose were detected as a table
and their sentences interleaved row-by-row. Telling the two apart needs a semantic signal
(whether the left and right cell's text are topically related) this module deliberately
doesn't have. A single column of text, or a table with genuinely data-shaped cells, is
unaffected.

**Headings.** Embedded PDF text carries real font metadata (size, bold) — a bold, larger-than-
body line becomes a `#`/`##`/`###` heading. OCR'd text has no font metadata to use, so it isn't
guessed at; OCR'd pages get reading-order/table structure only, not heading markup.

**Streaming.** A multi-page PDF/DOCX streams one event per page as each is ready. A single
image streams per detected text region as recognition completes — in arrival order (grouped by
aspect ratio for recognition accuracy, not top-to-bottom), with a final corrected, reading-order
result once every region is in. See `owistscribe.Scribe.stream()`'s docstring for the exact event
shapes — the web app and the Python API use the same ones.

## Speed and accuracy — what was tested, and what changed

**The detection resolution cap was costing real accuracy — raised.** Profiling a real tall test
image (1640×5666) against upstream RapidOCR's 2000px `max_side_len` cap showed confirmed-wrong
text from small label text shrunk past legibility by the forced downscale — not missed
detections, wrong characters. Measured at four resolutions:

| `max_side_len` | avg time | regions found |
|---|---|---|
| 2000 (upstream default) | 4.26s | 89 |
| **3000 (this project's default)** | **4.64s (+9%)** | **94** |
| 4000 | 6.45s (+51%) | 98 |
| 5666 (full res, no cap) | 10.00s (+135%) | 101 |

3000 recovers effectively all of the real accuracy fix for a modest +9% time cost; 4000/5666
mostly add a few marginal tokens at a much steeper cost. Overridable per-instance:
`Scribe(max_side_len=...)` / `ScribeEngine(max_side_len=...)`, or via the
`OWISTSCRIBE_MAX_SIDE_LEN` environment variable for the web app.

**Tested and genuinely didn't help, said plainly:**
- Forcing onnxruntime's thread count (either `1` or `os.cpu_count()`) — no measured win over
  its own `-1`/auto-detect default; `intra_op_num_threads=1` specifically nearly *doubled*
  inference time, real proof multi-threading matters, just not worth overriding further.
- `enable_cpu_mem_arena` — a ~1.7% difference, indistinguishable from run-to-run noise.
- Multi-page parallelism (thread pool, process pool) on 2-core test hardware — no measured gain;
  may look different on a many-core machine, but that's not what was tested.
- Content-cropping before detection — tested as a speed idea, shipped as an *accuracy* fix
  instead (`preprocess.py`'s `crop_to_content`), because that's what a real padded-canvas test
  actually showed: unchanged speed, but fewer recognition errors.

Full, dated, itemized record of every patch to the vendored engine (including two real bugs
found and fixed during development — a streaming-order bug and a crash on textless images):
`owistscribe/_core/CHANGES.md`.

## Development

```bash
git clone https://github.com/openwist/owistscribe
cd owistscribe
python -m venv .venv && source .venv/bin/activate        # Windows: .\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip                      # an old pip can fail to find PyMuPDF wheels
pip install -e ".[desktop]"
owistscribe web
python -m pytest tests                                    # needs: pip install pytest
```

On Windows PowerShell, if `Activate.ps1` is blocked, run `Set-ExecutionPolicy -Scope Process Bypass`
first, or skip activation and call `.\.venv\Scripts\owistscribe.exe` directly. If another copy of
`owistscribe` is installed globally, only the activated (or full-path) one is the one you edit.

## Web interface

```
owistscribe web          # then open http://127.0.0.1:8000
```

Drop files (or choose them, or paste an image) and reading starts by itself. Text streams in
line by line and the view follows the file being read; click another file and it stays there.
Stop cancels the run. Save lets you choose Markdown or plain text and rename the file before it is
created; several files can be saved as a zip or one combined file. The defaults already give the best
quality, so Settings only holds exceptions: Tables (Auto, Always, Off), spelling fix, and under More
options the noise filter, quality and how many files are read at once.

"Files read at once" defaults to what the computer can handle (each extra file in parallel costs
about 400 MB); on a 4 GB machine use 1.

## Speed, memory and GPU

`owistscribe info` shows what the engine will do on this computer; `owistscribe tune` measures
alternatives here and keeps one only if it is faster with identical text.

- **Memory.** Memory use is a short spike during detection, not a steady load. The engine
  chooses its detector size from free RAM (about 350 MB peak at 1280 px, 620 MB at 2000 px,
  1 GB at 3000 px, measured). With little RAM it re-reads tiny text in bands rather than
  dropping regions.
- **Detail.** `--detail fast` reads once and reads once. `balanced` (default) re-reads tiny text. `maximum` also re-reads uncertain lines and is slower.
  `maximum` always reads at the largest size memory allows.
- **GPU.** The default install uses the CPU. For an NVIDIA GPU:
  `pip uninstall onnxruntime` then `pip install onnxruntime-gpu`. For any DirectX 12 GPU on
  Windows: `pip uninstall onnxruntime` then `pip install onnxruntime-directml`. Then
  `owistscribe info` and `owistscribe ocr scan.png --device auto`. If the GPU cannot be used,
  OwistScribe falls back to CPU and says why. This path has not been tested on real GPU hardware.

## Many files at once

```bash
owistscribe scans/ --out-dir out                 # every image/PDF/DOCX in a folder
owistscribe a.png b.pdf 'photos/*.jpg' --out-dir out --format json
owistscribe scans/ --recursive --workers 2 --out-dir out
```

`--out-dir` writes one result file per input (`--format md|txt|json`) plus `batch_summary.json`.
A file that fails is reported and the others carry on (exit code 1 at the end). Each worker is a
separate process with its own models (~400MB), so the default worker count is capped by CPU and
free memory; on a 4GB machine leave it at the default or use `--workers 1`.

```python
from owistscribe import Scribe
items = Scribe().read_many(["scans/", "extra.pdf"], workers=2)
for it in items:
    print(it.path, "ok" if it.ok else it.error)   # it.result.text when ok
```

## Table and noise controls

```bash
owistscribe page.png --tables auto      # default: columns of ordinary prose are NOT turned into tables
owistscribe page.png --tables strict    # fewer tables; only short, data-like cells
owistscribe page.png --tables always    # geometry only (1.0 behaviour)
owistscribe page.png --tables off       # never render a table
owistscribe note.jpg --clean strict     # handwriting / noisy scans: drop low-confidence and symbol-heavy junk
```

Python: `Scribe(tables="auto", clean="auto")`. Web app: set `OWISTSCRIBE_TABLES` /
`OWISTSCRIBE_CLEAN` before `owistscribe web`.

`--spell en` (off by default) corrects words that are not real words but are one or two edits
from a common English word ("satistird" -> "satisfied"). It can also change valid words it does
not know (names, Swahili, technical terms) and sometimes picks the wrong word, so protect your
terms with `--dictionary words.txt` and check the output. `--clean` also drops scanner-app
watermarks such as "CamScanner".

What these do and don't do: the table check looks at the recognized cell text (short/numeric
cells = table, long sentence-like cells = columns of prose, read column by column). It is a
heuristic, not page understanding. If the detector merges two columns of a line into one wide box,
that line can't be split. `--clean` removes strings that are clearly not text (symbols only, one
character repeated, replacement/control characters); it cannot make the recognizer read real
handwriting correctly — that needs a recognizer trained on handwriting.

## License

Apache License 2.0 — see `LICENSE`. See [Credits](#credits) for third-party components.

## Credits

This package includes a modified fork of RapidOCR
(RapidAI, Apache-2.0; source in `owistscribe/_core/`, changes recorded there) running PaddleOCR's PP-OCR models, and, for the web app's
Markdown rendering, a copy of [marked](https://github.com/markedjs/marked)
(`owistscribe/static/marked.min.js`, MIT, Copyright MarkedJS/Christopher Jeffrey). See `NOTICE`
for the full attribution required by Apache-2.0 Section 4.

## What's verified and what isn't

Verified by running real files through real code and checking the actual output, not assumed:
CLI output for images/PDFs/DOCX, `--json` output shape, the web app's streaming and
non-streaming paths for every supported file type, the Markdown rendering in the browser, a
fresh `pip install` of the built wheel into a clean virtual environment (not the dev tree).

Not verified: GPU execution on real hardware, scaling on machines with many cores, and swapping behaviour on a machine with about 500 MB free (only peak memory was measured). Also not verified: a real build of the optional desktop app's packaged executable (PyInstaller
doesn't cross-compile, and this was built in a Linux-only sandbox) — said plainly rather than
implied. The desktop app itself runs fine from source (`owistscribe desktop`); only the "package it
into a standalone .exe/.app" step is unverified.
