Metadata-Version: 2.4
Name: document2md
Version: 0.3.0
Summary: Convert a PDF or a set of scanned page images to Markdown via OCR.
License: Apache-2.0
Project-URL: Homepage, https://github.com/INGEOTEC/document2md
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: mineru[pipeline]
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Dynamic: license-file

# document2md

[![test](https://github.com/INGEOTEC/document2md/actions/workflows/test.yml/badge.svg)](https://github.com/INGEOTEC/document2md/actions/workflows/test.yml)
[![Documentation Status](https://readthedocs.org/projects/document2md/badge/?version=latest)](https://document2md.readthedocs.io/en/latest/)

Converts a PDF, or a set of scanned page images, into Markdown — any
document, such as an edition of Mexico's official gazette (DOF, *Diario
Oficial de la Federación*) — optionally cropped down to a single note.
It's a wrapper
around [mineru](https://github.com/opendatalab/MinerU) for the OCR/layout
analysis itself; `document2md`'s own contribution is:

- Keeping mineru's `mineru-api` server warm across a batch of documents,
  instead of paying its startup (and model-loading) cost once per document.
- Stitching the OCR of a list of page images (several scanned pages of the
  same note) into one continuous Markdown document.
- Rewriting the raw HTML tables mineru falls back to (rowspan/colspan) into
  Markdown tables, so the output is Markdown all the way through.
- Cropping the result down to a single note, by locating its title and the
  next note's title in the OCR'd text — useful because a scanned page
  usually holds the tail of one note and the head of the next.

It was extracted from the [LegalIA](https://github.com/INGEOTEC/LegalIA)
monorepo at commit `e1f258c`, where every commit of its earlier history (as
`packages/document2md`, and as `packages/dof2md` before the rename) can still
be read.

## Install

```bash
pip install document2md
```

For development, from a clone of this repository:

```bash
pip install -e ".[test]"
```

## Usage

### CLI

`document2md` takes exactly one input source — a local PDF or a set of local page
images — and converts it to Markdown. It never downloads anything itself;
get the PDF first (e.g. `dofjson.download_edicion_pdf` for a whole DOF
edition by date and edition, see the
[dofjson README](https://github.com/INGEOTEC/LegalIA/tree/master/packages/dofjson)),
then convert it:

```bash
document2md --pdf edicion.pdf   # a local PDF

document2md --images pagina-1.jpg pagina-2.jpg \
    --filename out.md      # scanned pages, in order
```

`--filename` sets the output Markdown's name; with `--pdf` it defaults to
the PDF's own name (`edicion.pdf` → `edicion.md`), but with `--images` it's
required, since a set of images has no single name to derive one from.
`--outdir` sets the output directory (default: `output/`).

Since one edition's PDF holds every note published that day,
`--titulo`/`--titulo-siguiente` crop the resulting Markdown down to just one
note — its own title, and the next note's title, as they appear in the
gazette's own index:

```bash
document2md --pdf edicion.pdf \
    --titulo "ACUERDO por el que se..." \
    --titulo-siguiente "DECRETO por el que se..."
```

Title matching is fuzzy (OCR text rarely matches an index title exactly), so
a match below `--min-confidence` (default `0.6`) is treated as not found and
the crop falls back to keeping more text rather than dropping content. Other
flags:

- `--keep-pages` — also keep the uncropped Markdown, as
  `<outdir>/<pdf stem>.full.md`.
- `--keep-mineru-output` — keep mineru's own raw output (layout/model JSON,
  rendered PDFs...) in `<outdir>/<pdf stem>_mineru/` instead of discarding
  it; useful when a conversion looks wrong and mineru's own read of the page
  is the first thing worth inspecting.

### Python: batch conversion

Converting many documents in one run is where mineru's startup cost starts
to matter. `BatchConverter` keeps a single `mineru-api` server warm across
the whole batch instead of restarting it per document:

```python
from document2md import BatchConverter

jobs = [
    ("a.pdf", "output", "a.md"),
    (["b-p1.jpg", "b-p2.jpg"], "output", "b.md"),
]

with BatchConverter() as convert:
    for path_or_paths, outdir, filename in jobs:
        convert(path_or_paths, outdir, filename)
```

Each call takes a single PDF path, or a list of image paths for a document
spanning several scanned pages, and writes the result to `outdir/filename`.
The same `titulo`/`titulo_siguiente`, `min_confidence`, `keep_pages` and
`keep_mineru_output` options the CLI exposes are also its keyword
arguments — see `BatchConverter.__call__`'s docstring for the full
signature.

`nota2md.legal_provisions` accepts an already-`__enter__`'d `BatchConverter`
as its own `converter` parameter, so a batch of DOF legal provisions can
share the same warm server too.

## Tests

```bash
pytest -v
```
