Metadata-Version: 2.4
Name: tdf-converter
Version: 0.2.0
Summary: Convert documents (PDF, DOCX, XLSX, HTML, etc.) into Token-Dense Format (TDF), a token-optimized plain-text format with 100% distinct-content recall that cuts LLM token costs 28-66% compared to Markdown on structured documents.
Author-email: Ashish <meashumishra76@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Ashish
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/meashumishra/TDF-Token-Dense-Format-
Project-URL: Repository, https://github.com/meashumishra/TDF-Token-Dense-Format-
Project-URL: Issues, https://github.com/meashumishra/TDF-Token-Dense-Format-/issues
Keywords: llm,tokens,markdown,pdf,document-conversion,prompt-compression
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing :: Markup
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7
Requires-Dist: pymupdf>=1.24
Requires-Dist: python-docx>=1.1
Requires-Dist: openpyxl>=3.1
Requires-Dist: python-pptx>=0.6
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=5.0
Provides-Extra: bench
Requires-Dist: markitdown[all]>=0.1; extra == "bench"
Requires-Dist: pytest>=8; extra == "bench"
Dynamic: license-file

# TDF — Token-Dense Format

Convert documents into a compact, reversible representation for LLM context.

TDF removes the redundancy that document formats carry for humans — repeated table headers, per-page running headers, boilerplate clauses, markup scaffolding — and emits what's left in a structured, parseable form. On table- and boilerplate-heavy documents this cuts tokens roughly in half. On prose it does very little, and this README says exactly where the line falls.

```bash
pip install tdf-converter
tdf stats report.pdf
tdf convert report.pdf --to tdf -o report.tdf
```

---

## Where TDF helps, and where it doesn't

Measured against a Markdown rendering of the same parsed document, `o200k_base`, legend included. Source documents are third-party — none authored for this project.

| Document | Source | Tokens saved |
|---|---|---|
| `owid-co2-data.csv` (3k rows) | owid/co2-data | **53.5%** |
| `absolute_beginners.rst` | numpy/numpy | 8.7% |
| PEP 8 | python/peps | 5.6% |
| `deployment.md` | kubernetes/website | **2.1%** |

The pattern is consistent and worth stating plainly: **TDF's savings come from structural redundancy.** Wide tables with repeating values, repeated legal or policy boilerplate, running page headers, and slide chrome all compress well. Prose and code documentation do not, because Markdown is already close to token-optimal for them and there is little redundancy left to remove.

If your documents are mostly prose, TDF will not save you money. Use Markdown.

Against a raw CSV — which is what most people actually paste into a model — the tabular win is smaller than the Markdown comparison suggests:

| | vs Markdown table | vs raw CSV |
|---|---|---|
| `owid-co2-data.csv` | 53.5% | **15.0%** |
| `orders.csv` | 45.8% | 39.2% |

Both numbers are real. The Markdown comparison is the fair one if your pipeline renders tables to Markdown (most document converters do). The raw-CSV comparison is the fair one if you paste CSVs directly.

<details>
<summary>Synthetic benchmark corpus (higher numbers, but generated by this project)</summary>

`bench/make_samples.py` generates seven documents that deliberately span the four redundancy patterns TDF targets. These numbers are reproducible (`python bench/make_samples.py && python bench/benchmark.py`) but the corpus was constructed to contain the redundancy being removed, so treat them as an illustration of the mechanism rather than as evidence of real-world savings.

| File | Markdown | TDF | Saved |
|---|---|---|---|
| `operating_review.pdf` | 3,588 | 1,232 | 65.7% |
| `services_agreement.docx` | 1,945 | 1,031 | 47.0% |
| `orders.csv` | 16,982 | 9,196 | 45.8% |
| `sales_report.xlsx` | 20,505 | 11,296 | 44.9% |
| `handbook.html` | 4,732 | 2,660 | 43.8% |
| `quarterly_deck.pptx` | 1,559 | 977 | 37.3% |
| `runbook.md` | 1,355 | 978 | 27.8% |

The Markdown baseline is not a strawman: MarkItDown independently produces token counts within ~1% of this project's own Markdown renderer on six of the seven files.

</details>

**Tokenizer-independent.** Savings differ by less than 0.5pp between `o200k_base` and `cl100k_base` on every file tested. The gains come from removing text, not from exploiting one tokenizer's quirks.

---

## Fidelity

TDF is reversible. `tdf verify` parses the emitted output back into the document model and compares it against the original, reporting two numbers:

- **`distinct_recall`** — fraction of distinct content words recovered.
- **`occurrence_ratio`** — fraction of word *occurrences* recovered.

Both are reported because they measure different things and the second is lower:

| Document | `distinct_recall` | `occurrence_ratio` |
|---|---|---|
| `orders.csv` | 100.0% | 100.0% |
| `services_agreement.docx` | 100.0% | 100.0% |
| `owid-co2-data.csv` | 100.0% | 99.7% |
| `operating_review.pdf` | 100.0% | 97.2% |
| `handbook.html` | 100.0% | 93.7% |
| `quarterly_deck.pptx` | 100.0% | 93.1% |
| PEP 8 | **99.5%** | 95.6% |

**The occurrence gap is mostly intentional.** A running page header repeated across 40 pages is emitted once under `!R`, and the parser cannot know how many times to repeat it. The information ("this string ran on every page") is preserved; the repetition is not. No lossy elision is active by default in any of the rows above.

**The PEP 8 row is not intentional.** Eight distinct tokens are dropped, including identifiers appearing inside code samples. This is a known defect, not a design decision — see Known limitations.

### What these metrics do not measure

Both are bag-of-words comparisons. They are blind to ordering and to cell-to-row assignment. A table where two rows' values have been swapped scores 100% on both. **Do not treat a passing `tdf verify` as a guarantee of semantic correctness**, particularly for data where row alignment carries meaning.

---

## Skeleton mode and progressive loading

Beyond compression, TDF can emit a navigable map of a document instead of its contents.

```bash
tdf convert large_report.pdf --to skeleton     # section index, ~1-3% of full tokens
tdf expand 3 7 11                              # emit only the sections you want
```

With `--tier`, low-density regions are replaced by `!E` markers carrying an id, a kind, a token count and a one-line gist. An agent reads the map, decides what it needs, and requests regions by id:

```bash
tdf convert large_report.pdf --tier --to tdf   # elides index-like regions
tdf expand-elided x1                           # resolve one region
```

This is the pointer-not-payload pattern applied at document-region granularity. It is the most differentiated capability here — no other document converter offers it — and it is also the least validated. See Known limitations.

---

## Usage

```
tdf convert <file> [--to tdf|md|skeleton] [-o OUT] [--no-legend] [--tier] [--raw]
tdf diff    <old> <new>            # structural document diffing (50-99% token savings over raw comparison)
tdf stats   <file> [--json]        # token counts across all output modes
tdf verify  <file> [--json]        # round-trip fidelity report
tdf expand  <file> <section-ids>   # emit selected sections
tdf expand-elided <file> <id>      # resolve an !E region
tdf validate <file.tdf>            # check structural invariants
```

**Inputs:** `.pdf` `.docx` `.pptx` `.xlsx` `.xlsm` `.csv` `.tsv` `.html` `.md` `.txt` `.log`

**`--no-legend`** omits the ~130-token syntax legend from each document. Use it when you can place the legend in your system prompt once instead of paying for it per document — the difference matters in RAG, where retrieving 20 chunks otherwise means paying for the legend 20 times.

```python
from tdf.readers import read
from tdf.emit import render_tdf
from tdf.columnar import encode_columns

doc = read("report.pdf")
books = encode_columns(doc)          # must run on the same doc object
print(render_tdf(doc, codebooks=books))
```

---

## Format

```
!T <n> <caption>      table declaration          !D  phrase dictionary
!C <col> <col> ...    column header row          §n  dictionary reference
!V <col>              column value codebook      ^   repeat cell above
!R                    running header/footer       !P  page boundary
!F key=value          constant field             !E  elided region
```

A table with coded columns:

```
!V product
a Bearing Standard
b Coupler XL
!C order_id product region amount qty
10382 a g 74974 21
10315 a g 65256 38
```

Full grammar in [`docs/SPEC.md`](docs/SPEC.md). `tdf validate` checks any `.tdf` file against the structural invariants.

---

## How this compares

| | TDF | LLMLingua / LongLLMLingua | Gist / soft prompts | Docling / Marker / MinerU |
|---|---|---|---|---|
| Mechanism | structural redundancy removal | perplexity-based token pruning | learned embeddings | layout extraction |
| Reversible | yes | no | no | n/a |
| Needs a model | no | small LM | training + embedding access | ML layout models |
| Compression | ~1.5–2x on tables | 2–5x | up to 20–26x | n/a |
| Runtime | milliseconds, CPU | GPU-assisted | GPU | GPU/CPU |

**TDF is not a competitor to Docling, Marker or MinerU.** They extract structure from PDFs using trained layout models; TDF encodes an already-parsed document. Its built-in PDF reader is PyMuPDF-based heuristics and is not competitive with theirs on hard pages. Feeding TDF from one of those tools is the better pipeline: *they extract, TDF encodes.*

Against the prompt-compression family, TDF occupies a different point: deterministic, reversible, and usable through any API with no model access. It compresses less. It also composes — structural compression first, semantic compression on top, if you need both.

---

## Known limitations

**Accuracy impact is not yet measured.** This is the most important caveat in this README. TDF is validated for *token reduction* and *round-trip reversibility*. Whether an LLM answers questions as accurately from TDF as from Markdown has not been established.

There is reason for genuine concern. Published work on table serialization consistently finds that redundancy *helps* comprehension — formats that repeat column names per row let models read each row independently instead of cross-referencing distant headers, and they outperform denser formats despite costing more tokens. TDF's `!V` codebooks and `§n` dictionaries increase that cross-reference distance further than any format in those studies. An evaluation is in progress; until it lands, treat compression as the only validated benefit and test on your own workload before deploying.

**PEP 8 recall gap.** ~0.5% of distinct tokens are dropped on that document, concentrated in code samples. Under investigation.

**Small documents can grow.** The legend is a fixed ~130-token cost. On a short prose document — `bench/samples_tables/prose_only.pdf` gains 115 tokens — TDF is a net loss. Use `--no-legend`, or use Markdown.

**Fidelity metrics are order-blind.** See above.

**No OCR.** Scanned PDFs yield nothing. Use MinerU or Marker upstream.

**Elision honesty is unmeasured.** Whether models correctly *request* an elided region rather than fabricating an answer has not been quantified. Until it is, `--tier` should be considered experimental.

---

## Roadmap

Not yet shipped. Listed so the gap between claims and code stays visible.

- **Hybrid emission** — choose per block between Markdown and dense TDF, so output is never larger than Markdown and prose stays in its native form. This turns the range above into a floor guarantee.
- **Accuracy harness** — token-vs-accuracy Pareto frontier across formats and models, with ablations isolating `!V`, `§n` and `^`. Mechanisms that cost accuracy will be disabled by default regardless of their compression contribution.
- **`DoclingDocument` reader** — accept Docling output as input, inheriting its table detection and OCR.

---

## Development

```bash
git clone https://github.com/meashumishra/TDF.git
cd TDF
pip install -e ".[bench]"
python bench/make_samples.py        # required — without it 22 tests silently skip
python bench/make_table_pdfs.py
pytest tests/ -v
```

Expected: 88 passed, 3 skipped.

`tiktoken` downloads its BPE tables on first use; in a sandboxed environment set `TIKTOKEN_CACHE_DIR` to a pre-populated cache.

Issues and PRs: https://github.com/meashumishra/TDF

## License

MIT
