Metadata-Version: 2.5
Name: dot-parser
Version: 2.0.0
Summary: Document-to-markdown parser and chunker for RAG pipelines
Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
Project-URL: Issues, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/issues
Author-email: Kannon For Deep Tech <louis.letarnec@deepika.ai>
License-Expression: AGPL-3.0-or-later
License-File: LICENSE.md
Keywords: chunking,deepika,markdown,open-toolbox,parser,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.12
Requires-Dist: markitdown[all]>=0.1
Requires-Dist: pillow>=10.0
Requires-Dist: pydantic>=2
Requires-Dist: pymupdf4llm>=1.27.2.2
Requires-Dist: python-docx>=1.2.0
Requires-Dist: semchunk>=3.0
Requires-Dist: typing-extensions>=4.16
Provides-Extra: all
Requires-Dist: docling>=2.0; extra == 'all'
Requires-Dist: llama-cloud>=2.4; extra == 'all'
Requires-Dist: mistralai>=2.0; extra == 'all'
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == 'docling'
Provides-Extra: llama
Requires-Dist: llama-cloud>=2.4; extra == 'llama'
Provides-Extra: mistral
Requires-Dist: mistralai>=2.0; extra == 'mistral'
Description-Content-Type: text/markdown

# dot-parser

[![PyPI](https://img.shields.io/pypi/v/dot-parser)](https://pypi.org/project/dot-parser/)
![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)
[![Licence: AGPL v3](https://img.shields.io/badge/licence-AGPL--3.0--or--later-blue)](LICENSE.md)
[![Pipeline](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/badges/main/pipeline.svg)](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/pipelines)

**Turn documents into clean Markdown, then into retrieval-ready chunks.**

```python
from dot_parser import parse, chunk

markdown = parse("report.pdf")
chunks = chunk(markdown)

for c in chunks:
    print(c.section_path, len(c.content))
    # ['# Report', '## Methods', '### Analysis'] 842
```

## Why dot-parser

A RAG pipeline needs two things from a document: faithful text, and chunks that
keep their place in the document's structure. Most tools give you one or the
other — parsers stop at raw text, splitters assume you already have Markdown and
cut it blind to headings.

dot-parser does both in one step. It converts PDF, DOCX, PPTX, HTML, XLSX, CSV,
Markdown and plain text into clean Markdown, then splits that Markdown into
chunks carrying their heading hierarchy in `section_path` — so a chunk still
knows it came from *Report › Methods › Analysis*.

The default install stays light: heavy backends are optional extras, and you pick
per document whether to run locally or through a cloud OCR. See
[docs/DESIGN.md](docs/DESIGN.md) for how it compares to Docling, MarkItDown and
LangChain splitters.

## Features

- One `parse()` call for PDF, DOCX, PPTX, HTML, XLSX, CSV, Markdown and text
- Swappable PDF backends: local and fast, local and layout-aware, or cloud OCR
- Heading-aware chunking with `section_path` metadata
- Image extraction with per-image descriptions, from a VLM or from OCR annotations
- Batch PDF parsing through the Mistral Batch API (50% cheaper)
- Cost estimation before you send anything
- Light by default — heavy backends live behind extras

## Installation

```bash
pip install dot-parser

# With optional PDF backends (each ~50 MB - 1 GB):
pip install 'dot-parser[docling]'
pip install 'dot-parser[mistral]'
pip install 'dot-parser[llama]'
pip install 'dot-parser[all]'
```

Requires Python 3.12+.

## Quick start

```python
from dot_parser import parse, chunk

markdown: str = parse("report.pdf")
chunks: list[Chunk] = chunk(markdown)

for c in chunks:
    print(c.section_path, c.heading, len(c.content))
    # ["# Report", "## Methods", "### Analysis"], "### Analysis", 842
```

For a different PDF backend:

```python
from dot_parser import parse, parse_pdfs, Mistral, Docling

# single PDF, cloud OCR
md = parse("scanned.pdf", backend=Mistral())

# many PDFs, batched (50% off via Mistral Batch API)
mds = parse_pdfs(["a.pdf", "b.pdf", "c.pdf"], backend=Mistral())

# local layout-aware ML pipeline
md = parse("paper.pdf", backend=Docling())
```

## API

### `parse(source, format=None, backend=None) -> str`

Converts a document to Markdown.

- **source** -- file path (`str` or `Path`) or raw `bytes`
- **format** -- explicit format hint (e.g. `"pdf"`, `"docx"`). Required when source is `bytes`, otherwise inferred from the file extension.
- **backend** -- optional PDF backend (`Pymu`, `Docling`, `Mistral`, `Llama`). Defaults to `Pymu()`. Backends only apply to PDF input.

Supported formats: `.pdf`, `.md`, `.txt`, `.docx`, `.pptx`, `.html`, `.xhtml`, `.htm`, `.xlsx`, `.csv`

Raises `ParseError` on unsupported formats or conversion failures.

### `parse_pdfs(sources, backend=None) -> list[str | None]`

Converts a list of PDFs to Markdown, in input order. Backends that implement a `parse_pdfs` method (e.g. `Mistral` via the Batch API, 50% off) handle the whole list in a single optimized call. Otherwise falls back to looping `parse()` per source. Failed items are returned as `None` (no exception raised).

### `parse_with_images(source, format=None, *, backend=None, vlm=None, annotate_images=True, include_tables=True) -> ParseResult`

Converts a `.pdf`, `.docx` or `.pptx` and extracts its images alongside the
Markdown, keeping `![name](name)` anchors at each image's position. Each image is
described either by a VLM or by OCR annotations from the same call. See
[docs/IMAGE_EXTRACTION.md](docs/IMAGE_EXTRACTION.md) for the two available paths per format.

### Backends

- **`Pymu(ocr_fallback=True)`** -- pymupdf4llm, default. Fast and CPU-only. The built-in OCR fallback kicks in when an OCR engine (tesseract, rapidocr, paddleocr) is available. Set `ocr_fallback=False` to force the no-OCR code path (matches a runtime image without OCR shipped).
- **`Docling(force_full_page_ocr=True)`** -- IBM Docling, layout-aware ML pipeline + Tesseract OCR. Robust on image-only and broken-encoding PDFs. Install with `pip install dot-parser[docling]`.
- **`Mistral(api_key=None, model="mistral-ocr-4-0")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
- **`Llama(api_key=None, tier="cost_effective")`** -- LlamaCloud parsing API. Reads `LLAMA_CLOUD_API_KEY` from env. Tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`. Install with `pip install dot-parser[llama]`.

#### Mistral OCR enrichments

Three optional extras on the `Mistral(...)` constructor, applying to the `*_with_images` methods. All default to off, so existing callers see no change:

```python
from dot_parser import parse_with_images, Mistral

result = parse_with_images(
    "spec.pdf",
    backend=Mistral(
        extract_headers_footers=True,  # page furniture -> result.pages[i].header/.footer
        confidence_scores="page",  # or "word"   -> result.pages[i].*_confidence
        extract_captions=True,  # figure captions -> image.original_caption
    ),
)

for page in result.pages:
    print(page.page, page.average_confidence, page.minimum_confidence)
```

- `extract_headers_footers` moves running headers/footers out of the markdown and into `ParseResult.pages`, so repeated "page 4 of 27" noise stays out of retrieval chunks. It **removes that text from the markdown** -- the only one of the three that changes `result.markdown`.
- `confidence_scores` surfaces OCR's self-reported quality. **Calibrate any threshold on your own corpus -- absolute cutoffs do not transfer.** Measured over the 39 pages of the benchmark corpus (`native_simple`, `image_only`, `broken_encoding`), `average_confidence` sat at 0.985-0.989 and `minimum_confidence` at 0.23-0.46 on *every* document regardless of quality. `minimum_confidence` is the single worst region on the page, so it is low even on clean pages and is not a page-quality alarm by itself.
- `extract_captions` fills `ExtractedImage.original_caption` from the figure caption printed beside each image. Needs OCR 4+ **and** `mistralai>=2.8`; on older SDKs it logs a warning and leaves captions `None` rather than failing.

If the OCR model rejects any of these parameters, the call is retried once without them -- you get the markdown and images, minus the enrichments.

#### Caption-grounded image interpretation

Mistral OCR's inline annotations (`annotate_images=True`) are produced from the cropped image alone -- the annotating model sees no caption and no page text, which on domain-specific figures yields confidently wrong descriptions. `interpret_images()` replaces that pass with client-side VLM calls grounded in each figure's own caption and surrounding markdown:

```python
from dot_parser import parse_with_images, interpret_images, Mistral, MistralVLM

result = parse_with_images(
    "paper.pdf",
    backend=Mistral(extract_captions=True),
    annotate_images=False,  # skip the ungrounded inline pass
)
result = interpret_images(result, MistralVLM())
```

Same number of VLM calls as `annotate_images=True`, but each one carries the caption. On a figure-heavy biomechanics paper this turned "magnetic field components" / "fluid flow through a curved pipe" into accurate descriptions of the actual rib diagrams.

### `chunk(markdown, max_tokens=None, merge=True) -> list[Chunk]`

Splits Markdown into chunks with heading-hierarchy metadata.

- **markdown** -- Markdown string (typically output of `parse()`)
- **max_tokens** -- maximum tokens per chunk. Auto-computed from the 95th percentile of section sizes (clamped 64-512) when `None`.
- **merge** -- when `True` (default), merges small adjacent sections sharing the same parent heading to reduce fragmentation.

The chunking algorithm is described in [docs/DESIGN.md](docs/DESIGN.md).

### `Chunk`

```python
@dataclass(frozen=True)
class Chunk:
    content: str  # full text of the chunk (heading + body)
    heading: str | None  # the heading line, if any
    section_path: list[str]  # hierarchy of headings leading to this chunk
```

### `ParseError`

Raised by `parse()` when a document cannot be converted.

### Cost estimation

`cost_per_1k_pages(backend)` and `estimate_cost(backend, pages, batch=False)`
report what a cloud backend will cost, without instantiating it or needing an
API key.

## Stability

`dot-parser` follows semantic versioning: everything exported from the top-level
package is covered, anything underscore-prefixed is internal and may change in
any release. Public names are never removed without a deprecation period.

```toml
dependencies = ["dot-parser>=1.0,<2"]
```

See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

## Roadmap

- [ ] Table extraction as structured data, not just Markdown
- [ ] Streaming parse for very large documents
- [ ] Additional cloud OCR backends

## Documentation

| Document | Contents |
|---|---|
| [docs/DESIGN.md](docs/DESIGN.md) | Design rationale, backend choices, chunking algorithm |
| [docs/IMAGE_EXTRACTION.md](docs/IMAGE_EXTRACTION.md) | Image-extraction paths per format |
| [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) | Environment setup, tests, code style |
| [docs/VERSIONING.md](docs/VERSIONING.md) | Versioning, deprecation policy, how to depend on this package |
| [docs/PUBLISHING.md](docs/PUBLISHING.md) | Cutting a release |
| [CHANGELOG.md](CHANGELOG.md) | Release history |

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the DCO
sign-off requirement, the licensing terms that apply to contributions, and how to
submit a change.

## Licence

Copyright (C) 2026 Kannon For Deep Tech (deepika)

This software is distributed under the GNU Affero General Public License,
version 3 or later — see [LICENSE.md](LICENSE.md).

A commercial licence is available for use in proprietary environments.
Contact: louis.letarnec@deepika.ai
