Metadata-Version: 2.4
Name: paper-digest
Version: 0.1.1
Summary: AI-optimized arXiv paper reader. Converts academic papers into LLM-legible Markdown with structural annotations.
Project-URL: Homepage, https://github.com/jingxuan/paper-digest
Project-URL: Issues, https://github.com/jingxuan/paper-digest/issues
Author: Jingxuan
License-Expression: MIT
Keywords: ai,arxiv,llm,markdown,paper,research
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.9
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: click>=8.0
Requires-Dist: requests>=2.28
Requires-Dist: tqdm>=4.60
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# paper-digest

Convert arXiv papers to AI-optimized Markdown. Math-perfect, section-tagged, noise-free.

```bash
pip install paper-digest
paper-digest 2412.15185
```

## Why

Feeding a PDF to Claude/GPT costs 30k+ tokens per paper. Most of it is formatting noise. If you're screening 100 papers, that's 3M tokens wasted.

paper-digest converts arXiv papers to clean Markdown with:
- **LaTeX math preserved as-is** — `$\mathcal{L} = \sum_i \ell_i$`, not garbled MathML
- **Section tags** — `## [METHOD] 3.1 Our Approach` so your AI can skip to what matters
- **Noise removed** — no headers/footers, acknowledgements, or page numbers

A 30k-token PDF becomes ~10k chars of structured Markdown. Use `--sections abstract,method` to cut it to ~3k.

## Usage

### CLI

```bash
# Full paper
paper-digest 2412.15185

# Just abstract + method (saves ~74%)
paper-digest 2412.15185 --sections abstract,method

# JSON output
paper-digest 2412.15185 --format json

# Batch processing
paper-digest --batch ids.txt --output ./papers/

# Keep everything (don't trim acknowledgements/appendix)
paper-digest 2412.15185 --raw

# Prefer HTML path (faster, but math may have artifacts)
paper-digest 2412.15185 --prefer-html
```

### Python API

```python
from paper_digest import PaperDigest

pd = PaperDigest()

paper = pd.fetch("2412.15185")
print(paper.markdown)            # Full AI-optimized Markdown
print(paper.section("method"))   # Just the method section
print(paper.metadata)            # Title, authors, date, section tags

# Filter sections
paper = pd.fetch("2412.15185", sections=["abstract", "method"])

# Batch
results = pd.fetch_batch(["2412.15185", "2302.13971"])
```

### Input formats

All of these work:

```
2412.15185
2412.15185v2
hep-ph/0301012
https://arxiv.org/abs/2412.15185
https://arxiv.org/pdf/2412.15185.pdf
```

## How it works

```
arXiv ID → LaTeX source (default) or HTML → parse → tag sections → trim noise → Markdown
```

Two conversion paths, automatic fallback:

| Path | Coverage | Math quality | Speed |
|------|----------|-------------|-------|
| **LaTeX** (default) | 90%+ arXiv papers | Perfect (native LaTeX) | ~3s |
| **HTML** (fallback) | 2024+ papers | Good (occasional MathML artifacts) | ~2s |

Default is LaTeX-first because math formulas are kept verbatim from the source — no conversion, no loss. Use `--prefer-html` if you want speed over math fidelity.

## Output format

```markdown
---
arxiv_id: "2412.15185"
title: "Tiled Diffusion"
authors: ["Or Madar", "Ohad Fried"]
date: "2024-12-19"
sections: ["ABSTRACT", "INTRODUCTION", "METHOD", "EXPERIMENTS", "CONCLUSION", "REFERENCES"]
---

## [ABSTRACT] Abstract

Image tiling — the seamless connection of disparate images...

## [METHOD] 3 Method

Given a set of desired output images $\mathcal{I}=\{I_1, I_2, ..., I_N\}$...
```

### Section tags

| Tag | Matches |
|-----|---------|
| `ABSTRACT` | Abstract |
| `INTRODUCTION` | Introduction, Background, Motivation |
| `RELATED_WORK` | Related Work, Prior Work, Literature Review |
| `METHOD` | Method, Methodology, Our Approach, Model Architecture, ... |
| `EXPERIMENTS` | Experiments, Experimental Setup, Evaluation, ... |
| `RESULTS` | Results, Main Results, Analysis, Discussion |
| `CONCLUSION` | Conclusion, Summary, Future Work |
| `REFERENCES` | References, Bibliography |
| `OTHER` | Anything else (original title preserved) |

## Motivation

> Transforming human knowledge from human-first and human-legible to LLM-first and LLM-legible is a beautiful space with so much potential... This would be a significantly richer source of legible, workable information for an LLM than just something like pdf-to-text (current prevailing practice), which simply asks the LLM to predict the textbook content top to bottom token by token (umm - lame).
>
> — **Andrej Karpathy**, [2025-08-28](https://x.com/karpathy/status/1961128638725923119)

## License

MIT
