Metadata-Version: 2.4
Name: nemo-retriever
Version: 2026.8.14.dev183
Summary: A modern RAG ingestion pipeline from Nvidia
Author-email: Jeremy Dyer <jdyer@nvidia.com>
License-Expression: Apache-2.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: backoff
Requires-Dist: ray[data,serve]>=2.56.1
Requires-Dist: ffmpeg-python
Requires-Dist: pandas<3,>=2.0
Requires-Dist: sqlglot>=30.0.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: typer>=0.12.0
Requires-Dist: click>=8.2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: jinja2>=3.1
Requires-Dist: fastapi>=0.114.0
Requires-Dist: uvicorn[standard]>=0.30.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: prometheus-fastapi-instrumentator<9,>=8.0
Requires-Dist: opentelemetry-api>=1.41.1
Requires-Dist: opentelemetry-sdk>=1.41.1
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.41.1
Requires-Dist: fastmcp>=2.0.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: requests>=2.33.0
Requires-Dist: urllib3==2.7.0
Requires-Dist: pydantic>=2.8.0
Requires-Dist: rich>=13.7.0
Requires-Dist: universal-pathlib>=0.2.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: debugpy>=1.8.0
Requires-Dist: fsspec>=2025.5.1
Requires-Dist: s3fs>=2025.5.1
Requires-Dist: pypdfium2==4.30.0
Requires-Dist: pillow>=12.3.0
Requires-Dist: nltk>=3.10.1
Requires-Dist: markitdown
Requires-Dist: langchain-nvidia-ai-endpoints>=1.4.0
Requires-Dist: lancedb
Requires-Dist: nvidia-riva-client>=2.25.1
Requires-Dist: tokenizers<0.23,>=0.21.1
Requires-Dist: huggingface-hub>=0.34.0
Provides-Extra: service
Requires-Dist: backoff; extra == "service"
Requires-Dist: litellm<2,>=1.95.0rc3; extra == "service"
Requires-Dist: easydict; extra == "service"
Requires-Dist: addict; extra == "service"
Requires-Dist: scikit-learn>=1.6.0; extra == "service"
Requires-Dist: psutil>=5.9.0; extra == "service"
Requires-Dist: apscheduler>=3.10; extra == "service"
Requires-Dist: librosa>=0.10.2; extra == "service"
Requires-Dist: numba>=0.59; extra == "service"
Provides-Extra: local
Requires-Dist: transformers<6,>=5.14.1; extra == "local"
Requires-Dist: accelerate==1.12.0; extra == "local"
Requires-Dist: opencv-python-headless>=4.8.0; extra == "local"
Requires-Dist: torch==2.11.0; sys_platform == "linux" and extra == "local"
Requires-Dist: torch==2.11.0; sys_platform == "win32" and extra == "local"
Requires-Dist: torch==2.11.0; sys_platform == "darwin" and extra == "local"
Requires-Dist: torchvision==0.26.0; sys_platform == "linux" and extra == "local"
Requires-Dist: torchvision==0.26.0; sys_platform == "win32" and extra == "local"
Requires-Dist: torchvision==0.26.0; sys_platform == "darwin" and extra == "local"
Requires-Dist: tritonclient; extra == "local"
Requires-Dist: einops; extra == "local"
Requires-Dist: easydict; extra == "local"
Requires-Dist: addict; extra == "local"
Requires-Dist: scikit-learn>=1.6.0; extra == "local"
Requires-Dist: timm==1.0.22; extra == "local"
Requires-Dist: albumentations==2.0.8; extra == "local"
Requires-Dist: nemotron-page-elements-v3<4,>=3.0.1; extra == "local"
Requires-Dist: nemotron-table-structure-v1<2,>=1.0.0; extra == "local"
Requires-Dist: nemotron-ocr<3,>=2.0.0; (sys_platform == "linux" and (platform_machine == "x86_64" or platform_machine == "aarch64")) and extra == "local"
Requires-Dist: nvidia-ml-py; extra == "local"
Requires-Dist: apscheduler>=3.10; extra == "local"
Requires-Dist: psutil>=5.9.0; extra == "local"
Requires-Dist: vllm==0.25.1; sys_platform == "linux" and extra == "local"
Requires-Dist: flashinfer-cubin==0.6.13; sys_platform == "linux" and extra == "local"
Requires-Dist: flashinfer-python==0.6.13; sys_platform == "linux" and extra == "local"
Provides-Extra: multimedia
Requires-Dist: soundfile>=0.12.0; extra == "multimedia"
Requires-Dist: scipy>=1.11.0; extra == "multimedia"
Requires-Dist: cairosvg>=2.7.0; extra == "multimedia"
Requires-Dist: librosa>=0.10.2; extra == "multimedia"
Requires-Dist: numba>=0.59; extra == "multimedia"
Provides-Extra: nemotron-parse
Requires-Dist: open-clip-torch==3.2.0; extra == "nemotron-parse"
Provides-Extra: tabular
Requires-Dist: duckdb>=1.2.0; extra == "tabular"
Requires-Dist: duckdb-engine>=0.13.0; extra == "tabular"
Requires-Dist: neo4j>=5.0; extra == "tabular"
Requires-Dist: langgraph>=1.2.0; extra == "tabular"
Provides-Extra: benchmarks
Requires-Dist: datasets>=4.8.2; extra == "benchmarks"
Requires-Dist: open-clip-torch==3.2.0; extra == "benchmarks"
Provides-Extra: llm
Requires-Dist: litellm<2,>=1.95.0rc3; extra == "llm"
Provides-Extra: dev
Requires-Dist: build>=1.2.2; extra == "dev"
Requires-Dist: pytest>=8.0.2; extra == "dev"
Provides-Extra: all
Requires-Dist: nemo_retriever[benchmarks,llm,local,multimedia,nemotron-parse,service,tabular]; extra == "all"
Dynamic: license-file

# Quick Start for NeMo Retriever Library

NeMo Retriever Library is a retrieval-augmented generation (RAG) ingestion pipeline for documents that can parse text, tables, charts, and infographics. NeMo Retriever Library parses documents, creates embeddings, optionally stores embeddings in LanceDB, and performs recall evaluation.

This quick start guide shows how to run NeMo Retriever Library as a library in local Python processes without containers. Choose one inference path:

- **Local GPU (Linux):** Pull and run [Nemotron RAG models from Hugging Face](https://huggingface.co/collections/nvidia/nemotron-rag) on your GPU(s). Requires CUDA 13.x and the `[local]` extra.
- **Remote NIM:** Call build.nvidia.com hosted or self-hosted NeMo Retriever NIM endpoints over the network. The base package installs on Linux, Windows x64, and macOS Apple Silicon (arm64); no local GPU is required. macOS Intel (x86_64) is not supported.

The steps below cover environment setup, installation, and a first ingestion run. For Kubernetes or container deployments, refer to [Deployment at a glance](#deployment-at-a-glance) and the [Pre-Requisites & Support Matrix](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/).

## Deployment at a glance

For Kubernetes deployments, use the **[`nemo_retriever/helm` chart](helm/README.md)** to deploy the retriever **service** and optional in-cluster **NIM** workloads. Published Helm install and upgrade flows for the full extraction stack are documented in the **[NeMo Retriever Library](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/)**; use those docs together with the chart README for your release.

For standalone service-image builds and local container runs, see **[`docker.md`](docker.md)**.

## Prerequisites

Before starting, confirm requirements for your inference path (refer to the [Pre-Requisites & Support Matrix](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/)):

- **Local GPU inference (Linux):** CUDA 13.x with `libcudart.so.13` available, and GPUs visible to the system.
- **Remote NIM inference:** Python 3.12 and network access to your NIM endpoints; CUDA is not required on the client host.

If optical character recognition (OCR) fails with a `libcudart.so.13` error on a local GPU path, install the CUDA 13 runtime for your platform and update `LD_LIBRARY_PATH` to include the CUDA lib64 directory, then rerun the pipeline.

For example, the following command can be used to update the `LD_LIBRARY_PATH` value.

```bash
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/lib64
```

## Setup your environment

Complete the following steps to set up your environment. You will create and activate isolated Python and project virtual environments, install the NeMo Retriever Library and its dependencies, and then run the provided ingestion snippets to validate your setup.

1. Create and activate the NeMo Retriever Library environment

Before installing NeMo Retriever Library, create an isolated Python environment so its dependencies do not conflict with other projects on your system. In this step, you set up a new virtual environment and activate it so that all subsequent installs are scoped to NeMo Retriever Library.

In your terminal, run the following commands from any location.

**Local GPU (Linux)**

Install with the `[local]` extra, which includes Nemotron model packages, transformers, and GPU tooling:

```bash
uv venv retriever --python 3.12
source retriever/bin/activate
uv pip install "nemo-retriever[local]"
```

The `[local]` extra resolves stable Nemotron extraction packages by default. To
try prerelease/nightly Nemotron packages from PyPI within the same supported
major-version windows, opt in with `--pre`:

```bash
uv pip install --pre "nemo-retriever[local]==26.08-RC1"
```

**Remote NIM (no local GPU)**

The base package is sufficient when all inference runs through hosted or self-hosted NIM endpoints:

```bash
uv python install 3.12
uv venv retriever --python 3.12
source retriever/bin/activate
uv pip install nemo-retriever
```

Install matching **ingestion client** and **ingestion runtime** wheels at the same version when your workflow expects them (refer to the [NeMo Retriever Library prerequisites](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) for the exact PyPI coordinates for your release).

This creates a dedicated Python environment and installs the `nemo-retriever` PyPI package, the canonical distribution for the NeMo Retriever Library. The base install includes the lightweight tokenizer dependencies used for TXT/HTML chunking (no Transformers or local model weights).

If your PDF pipeline uses `method="nemotron_parse"`, install the Nemotron Parse client dependencies with the `nemotron-parse` extra:

```bash
uv pip install "nemo-retriever[nemotron-parse]"
```

For local GPU inference with Nemotron Parse, combine the extras as `nemo-retriever[local,nemotron-parse]`.

> **Note:** `uv python install 3.12` installs a uv-managed Python that includes development headers (`Python.h`). These headers are required by vLLM, which compiles CUDA kernels at runtime using torch inductor. If you skip this step and use a system Python without headers, vLLM actor initialization will fail with `InductorError: fatal error: Python.h: No such file or directory`.

2. Override Torch and Torchvision with CUDA 13 builds (local GPU only)

The `[local]` extra pulls PyTorch from PyPI, which defaults to a CPU build on Linux. Reinstall from the CUDA 13.0 wheel index to match the CUDA runtime required by the Nemotron model packages:

```bash
uv pip uninstall torch torchvision
uv pip install torch==2.11.0 torchvision==0.26.0 -i https://download.pytorch.org/whl/cu130
```

Skip this step if you are using remote NIM inference only.

## Run the pipeline

The [test PDF](../data/multimodal_test.pdf) contains text, tables, charts, and images. Additional test data resides [here](../data/).

> **Note:** `retriever ingest` defaults to local, in-process execution. Use `retriever ingest batch ...` for Ray Data scale-out on larger workloads.
> File formats and internal extraction stages are not separate root commands; configure supported behavior through `retriever ingest`.

The examples below use default local GPU inference (no `invoke_url` specified) and require the `[local]` extra and the CUDA 13 torch override from the setup steps above. For remote NIM inference without a local GPU, refer to [Run with remote inference](#run-with-remote-inference-no-local-gpu-required).

### Ingest a test pdf
```python
from nemo_retriever import create_ingestor
from nemo_retriever.common.io import to_markdown, to_markdown_by_page
from pathlib import Path

documents = [str(Path("../data/multimodal_test.pdf"))]
ingestor = create_ingestor(run_mode="batch")

# ingestion tasks are chainable and defined lazily
ingestor = (
  ingestor.files(documents)
  .extract(
    # below are the default values, but content types can be controlled
    extract_text=True,
    extract_charts=True,
    extract_tables=True,
    extract_infographics=True
  )
  .embed()
  .vdb_upload()
)
```

### Ingest inline text

Python callers can pass raw text documents directly to the same text splitting,
embedding, and vector database graph without creating temporary files. Inline
text is supported in `inprocess`, `batch`, and `service` run modes.

```python
from nemo_retriever import create_ingestor

texts = ["some text", "another longer text"]

chunks = (
  create_ingestor(run_mode="batch")
  .texts(texts)
  .embed()
  .vdb_upload()
  .ingest()
)
```

Each string is treated as a raw document and split with `TextChunkParams`
defaults. To override chunking for every text source in the ingest, add
`.extract(split_config={"text": {"max_tokens": 512, "overlap_tokens": 64}})`.
Inline text can be combined with
`.files(...)` and, in modes that support them, `.buffers(...)`; each source is
routed through its matching extractor before the results enter the shared
embedding and sink stages. Inline corpora remain resident in client or driver
memory, so prefer file ingestion when the corpus may exceed the available memory.

### Optional extras

- **`multimedia`** — Audio/video extraction and SVG rendering support. Install this extra when using Parakeet ASR through `extract_method="audio"` so audio decoding and resampling dependencies are available:
  ```bash
  uv pip install "nemo-retriever[multimedia]"
  # or, for local GPU inference:
  uv pip install "nemo-retriever[local,multimedia]"
  ```

Run the batch pipeline script and point it at the directory that contains your PDFs using the following command.

```bash
uv run python nemo_retriever/src/nemo_retriever/examples/batch_pipeline.py /path/to/pdfs
```

```python
# ingestor.ingest() actually executes the pipeline
chunks = ingestor.ingest()  # pandas.DataFrame (batch and inprocess)
```

### Ingest a test corpus (CLI)

`retriever ingest` accepts a file or a directory and writes a ready-to-query
LanceDB table. From a clone of this repository, `./data/multimodal_test.pdf`
is a valid first-run input. Replace `/path/to/file-or-directory` below with
that PDF or with your own file or directory.

> **Small corpora are supported.** The NeMo Retriever LanceDB adapter requests
> 16 IVF partitions by default. When the table has fewer rows, the adapter
> clamps that count to one less than the row count. An ingest that produces
> 2 through 15 rows still builds an IVF index. A one-row table is stored
> without a vector index, and ingest succeeds. An ingest that produces zero rows
> is a separate empty-result error. Use a larger corpus when you want more
> representative retrieval quality.

```bash
retriever ingest /path/to/file-or-directory \
  --lancedb-uri lancedb \
  --table-name nemo-retriever
```

Chunks land at `./lancedb/nemo-retriever`, which matches the storage settings
used in [Run a recall query](#run-a-recall-query) below. With the
`[local]` extra installed (see setup), defaults point at local-GPU extraction
and embedding.

**No local GPU?** Set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) (refer to [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)) and route extraction and embedding
through [build.nvidia.com](https://build.nvidia.com/) NIMs instead:

```bash
export NVIDIA_API_KEY=nvapi-...

retriever ingest /path/to/file-or-directory \
  --lancedb-uri lancedb \
  --table-name nemo-retriever \
  --page-elements-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3 \
  --ocr-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v2 \
  --table-structure-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1 \
  --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2
```

> **OCR engine default:** The default OCR engine is **Nemotron OCR v2**. Use
> `--ocr-version v1` to opt into the legacy OCR engine. Local OCR v2 defaults
> to multilingual mode (`multi`); pass `--ocr-lang english` for the English-only
> v2 selector. Remote OCR NIM endpoints decide their own model and language
> behavior, and the local OCR selectors are not added to remote request payloads.

When you use a remote embedder, the endpoint and provider prefix remain runtime
configuration. The query model is read from LanceDB metadata when available;
pass an explicit model only for an override or a legacy table without metadata.

### Inspect extracts
You can inspect how recall accuracy optimized text chunks for various content types were extracted into text representations:
```text
# page 1 raw text:
>>> chunks.iloc[0]["text"]
'TestingDocument\r\nA sample document with headings and placeholder text\r\nIntroduction\r\nThis is a placeholder document that can be used for any purpose...'

# markdown formatted table from the first page
'| Table | 1 |\n| This | table | describes | some | animals, | and | some | activities | they | might | be | doing | in | specific |\n| locations. |\n| Animal | Activity | Place |\n| Giraffe | Driving | a | car | At | the | beach |\n| Lion | Putting | on | sunscreen | At | the | park |\n| Cat | Jumping | onto | a | laptop | In | a | home | office |\n| Dog | Chasing | a | squirrel | In | the | front | yard |\n| Chart | 1 |'

# a chart from the first page
>>> chunks.iloc[2]["text"]
'Chart 1\nThis chart shows some gadgets, and some very fictitious costs.\nGadgets and their cost\n$160.00\n$140.00\n$120.00\n$100.00\nDollars\n$80.00\n$60.00\n$40.00\n$20.00\n$-\nPowerdrill\nBluetooth speaker\nMinifridge\nPremium desk fan\nHammer\nCost'

# markdown formatting for full pages or documents:
# per-page markdown is keyed by page number
>>> to_markdown_by_page(chunks).keys()
dict_keys([1, 2, 3])

>>> to_markdown_by_page(chunks)[1]
'TestingDocument\r\nA sample document with headings and placeholder text\r\nIntroduction\r\nThis is a placeholder document that can be used for any purpose. It contains some \r\nheadings and some placeholder text to fill the space. The text is not important and contains \r\nno real value, but it is useful for testing. Below, we will have some simple tables and charts \r\nthat we can use to confirm Ingest is working as expected.\r\nTable 1\r\nThis table describes some animals, and some activities they might be doing in specific \r\nlocations.\r\nAnimal Activity Place\r\nGira@e Driving a car At the beach\r\nLion Putting on sunscreen At the park\r\nCat Jumping onto a laptop In a home o@ice\r\nDog Chasing a squirrel In the front yard\r\nChart 1\r\nThis chart shows some gadgets, and some very fictitious costs.\n\n| This | table | describes | some | animals, | and | some | activities | they | might | be | doing | in | specific |\n| locations. |\n| Animal | Activity | Place |\n| Giraffe | Driving | a | car | At | the | beach |\n| Lion | Putting | on | sunscreen | At | the | park |\n| Cat | Jumping | onto | a | laptop | In | a | home | office |\n| Dog | Chasing | a | squirrel | In | the | front | yard |\n| Chart | 1 |\n\nChart 1 This chart shows some gadgets, and some very fictitious costs. Gadgets and their cost $160.00 $140.00 $120.00 $100.00 Dollars $80.00 $60.00 $40.00 $20.00 $- Powerdrill Bluetooth speaker Minifridge Premium desk fan Hammer Cost\n\n### Table 1\n\n| This | table | describes | some | animals, | and | some | activities | they | might | be | doing | in | specific |\n| locations. |\n| Animal | Activity | Place |\n| Giraffe | Driving | a | car | At | the | beach |\n| Lion | Putting | on | sunscreen | At | the | park |\n| Cat | Jumping | onto | a | laptop | In | a | home | office |\n| Dog | Chasing | a | squirrel | In | the | front | yard |\n| Chart | 1 |\n\n### Chart 1\n\nChart 1 This chart shows some gadgets, and some very fictitious costs. Gadgets and their cost $160.00 $140.00 $120.00 $100.00 Dollars $80.00 $60.00 $40.00 $20.00 $- Powerdrill Bluetooth speaker Minifridge Premium desk fan Hammer Cost\n\n### Table 2\n\n| This | table | describes | some | animals, | and | some | activities | they | might | be | doing | in | specific |\n| locations. |\n| Animal | Activity | Place |\n| Giraffe | Driving | a | car | At | the | beach |\n| Lion | Putting | on | sunscreen | At | the | park |\n| Cat | Jumping | onto | a | laptop | In | a | home | office |\n| Dog | Chasing | a | squirrel | In | the | front | yard |\n| Chart | 1 |\n\n### Chart 2\n\nChart 1 This chart shows some gadgets, and some very fictitious costs. Gadgets and their cost $160.00 $140.00 $120.00 $100.00 Dollars $80.00 $60.00 $40.00 $20.00 $- Powerdrill Bluetooth speaker Minifridge Premium desk fan Hammer Cost\n\n### Table 3\n\n| This | table | describes | some | animals, | and | some | activities | they | might | be | doing | in | specific |\n| locations. |\n| Animal | Activity | Place |\n| Giraffe | Driving | a | car | At | the | beach |\n| Lion | Putting | on | sunscreen | At | the | park |\n| Cat | Jumping | onto | a | laptop | In | a | home | office |\n| Dog | Chasing | a | squirrel | In | the | front | yard |\n| Chart | 1 |\n\n### Chart 3\n\nChart 1 This chart shows some gadgets, and some very fictitious costs. Gadgets and their cost $160.00 $140.00 $120.00 $100.00 Dollars $80.00 $60.00 $40.00 $20.00 $- Powerdrill Bluetooth speaker Minifridge Premium desk fan Hammer Cost'

# full document markdown is a single string (or None if empty)
>>> to_markdown(chunks)[:50]
'# Extracted Content\n\n## Page 1\n\nTestingDocument\r\nA s'
```

Since the ingestion job automatically populated a lancedb table with all these chunks, you can use queries to retrieve semantically relevant chunks for feeding directly into an LLM:

### Run a recall query

```python
from nemo_retriever.graph.retriever import Retriever

retriever = Retriever(
  # values used by the retriever ingest example above
  vdb_kwargs={"uri": "lancedb", "table_name": "nemo-retriever"},
  top_k=5,
  rerank=False
)

query = "Given their activities, which animal is responsible for the typos in my documents?"

# you can also submit a list with retriever.queries[...]
hits = retriever.query(query)
```

If you ingested with the remote-NIM recipe above (no local GPU), point the
`Retriever` at the same embedding endpoint so query vectors are produced by the
same model that produced the stored chunk vectors:

```python
retriever = Retriever(
    vdb_kwargs={"uri": "lancedb", "table_name": "nemo-retriever"},
    embed_kwargs={
        "model_name": "nvidia/llama-nemotron-embed-1b-v2",
        "embed_model_name": "nvidia/llama-nemotron-embed-1b-v2",
        "embedding_endpoint": "https://integrate.api.nvidia.com/v1/embeddings",
    },
    top_k=5,
    rerank=False,
)
hits = retriever.query(query)
```

```text
# retrieved text from the first page
>>> hits[0]
{'text': 'TestingDocument\r\nA sample document with headings and placeholder text\r\nIntroduction\r\nThis is a placeholder document that can be used for any purpose. It contains some \r\nheadings and some placeholder text to fill the space. The text is not important and contains \r\nno real value, but it is useful for testing. Below, we will have some simple tables and charts \r\nthat we can use to confirm Ingest is working as expected.\r\nTable 1\r\nThis table describes some animals, and some activities they might be doing in specific \r\nlocations.\r\nAnimal Activity Place\r\nGira@e Driving a car At the beach\r\nLion Putting on sunscreen At the park\r\nCat Jumping onto a laptop In a home o@ice\r\nDog Chasing a squirrel In the front yard\r\nChart 1\r\nThis chart shows some gadgets, and some very fictitious costs.', 'metadata': '{"page_number": 1, "pdf_page": "multimodal_test_1", "page_elements_v3_num_detections": 9, "page_elements_v3_counts_by_label": {"table": 1, "chart": 1, "title": 3, "text": 4}, "ocr_table_detections": 1, "ocr_chart_detections": 1, "ocr_infographic_detections": 0}', 'source': '{"source_id": "/home/dev/projects/NeMo-Retriever/data/multimodal_test.pdf"}', 'page_number': 1, '_distance': 1.5822279453277588}

# retrieved text of the table from the first page
>>> hits[1]
{'text': '| Table | 1 |\n| This | table | describes | some | animals, | and | some | activities | they | might | be | doing | in | specific |\n| locations. |\n| Animal | Activity | Place |\n| Giraffe | Driving | a | car | At | the | beach |\n| Lion | Putting | on | sunscreen | At | the | park |\n| Cat | Jumping | onto | a | laptop | In | a | home | office |\n| Dog | Chasing | a | squirrel | In | the | front | yard |\n| Chart | 1 |', 'metadata': '{"page_number": 1, "pdf_page": "multimodal_test_1", "page_elements_v3_num_detections": 9, "page_elements_v3_counts_by_label": {"table": 1, "chart": 1, "title": 3, "text": 4}, "ocr_table_detections": 1, "ocr_chart_detections": 1, "ocr_infographic_detections": 0}', 'source': '{"source_id": "/home/dev/projects/NeMo-Retriever/data/multimodal_test.pdf"}', 'page_number': 1, '_distance': 1.614684820175171}
```

###  Generate a query answer using an LLM
The above retrieval results are often feedable directly to an LLM for answer generation.

To do so, first install the openai client and set your [build.nvidia.com](https://build.nvidia.com/) API key:
```bash
uv pip install openai
export NVIDIA_API_KEY=nvapi-...
```

```python
from openai import OpenAI
import os

client = OpenAI(
  base_url = "https://integrate.api.nvidia.com/v1",
  api_key = os.environ.get("NVIDIA_API_KEY")
)

hit_texts = [hit["text"] for hit in hits]
prompt = f"""
Given the following retrieved documents, answer the question: {query}

Documents:
{hit_texts}
"""

completion = client.chat.completions.create(
  model="nvidia/nemotron-3-super-120b-a12b",
  messages=[{"role":"user","content":prompt}],
  stream=False
)

answer = completion.choices[0].message.content
print(answer)
```

Answer:
```
Cat is the animal whose activity (jumping onto a laptop) matches the location of the typos, so the cat is responsible for the typos in the documents.
```

### Run agentic retrieval

Agentic retrieval runs an LLM-driven ReAct loop over an existing LanceDB index.
It does not ingest documents. Build the index with one of the ingestion flows
above, then query the same `lancedb_uri`, `table_name`, and embedding model.

By default, the agent LLM runs in process with local vLLM and `nemotron-8b`
(`nvidia/Llama-3.1-Nemotron-Nano-8B-v1`). This requires a CUDA GPU host and the
`[local]` extra. GPU placement follows process-level vLLM behavior, so set
`CUDA_VISIBLE_DEVICES` before starting the command. Embedding follows the same
local/remote split as dense retrieval; pass `--embed-invoke-url` to use a remote
embedding endpoint.

**Local in-process vLLM agent LLM.** Omit `--agentic-invoke-url` to load the
supported local agent LLM directly in the Python process. `nemotron-8b` is the
default; `super-49b` is also supported when the process has enough visible GPUs.
For `super-49b`, set `--agentic-local-tensor-parallel-size 2` with two visible
GPUs (for example `CUDA_VISIBLE_DEVICES=0,1`).

```bash
CUDA_VISIBLE_DEVICES=0 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
  --agentic \
  --lancedb-uri lancedb \
  --table-name nemo-retriever \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2
```

```bash
CUDA_VISIBLE_DEVICES=0,1 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
  --agentic \
  --agentic-llm-model super-49b \
  --agentic-local-tensor-parallel-size 2 \
  --lancedb-uri lancedb \
  --table-name nemo-retriever \
  --embed-model-name nvidia/llama-nemotron-embed-vl-1b-v2
```
**OpenAI-compatible agent endpoint.** Pass `--agentic-invoke-url` when you want a
custom model or a separately hosted chat-completions server, such as vLLM server
mode or a self-hosted NIM. When an invoke URL is provided, `--agentic-llm-model`
is required and is sent as the remote model ID.

```bash
retriever query "What is RAG?" \
  --agentic \
  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
  --agentic-invoke-url http://localhost:9000/v1/chat/completions \
  --lancedb-uri lancedb \
  --table-name nemo-retriever \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2
```

Unlike dense retrieval, agentic mode returns ranked document IDs as JSON, not
text-enriched hits.

For a quick smoke test, reduce agent work:

```bash
CUDA_VISIBLE_DEVICES=0 retriever query "What is RAG?" \
  --agentic \
  --lancedb-uri lancedb \
  --table-name nemo-retriever \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \
  --top-k 1 \
  --agentic-react-max-steps 1
```

You can run the same flow from Python. Omit `invoke_url` for the default local
in-process vLLM backend, or pass `invoke_url` on `QueryAgenticOptions` for a
separate OpenAI-compatible chat-completions endpoint.

```python
from nemo_retriever.cli.query_workflow import agentic_query_documents
from nemo_retriever.query.options import (
    QueryAgenticOptions,
    QueryEmbedOptions,
    QueryRequest,
    QueryRetrievalOptions,
    QueryStorageOptions,
)

# Local in-process vLLM: set CUDA_VISIBLE_DEVICES before starting Python.
# Remote agent LLM: pass invoke_url="http://localhost:9000/v1/chat/completions".
results = agentic_query_documents(
    QueryRequest(
        query="What is RAG?",
        retrieval=QueryRetrievalOptions(top_k=10),
        storage=QueryStorageOptions(
            lancedb_uri="lancedb",
            table_name="nemo-retriever",
        ),
        embed=QueryEmbedOptions(
            embed_model_name="nvidia/llama-nemotron-embed-1b-v2",
        ),
        agentic=QueryAgenticOptions(
            enabled=True,
            llm_model="nemotron-8b",
            local_llm_backend="vllm",
            local_gpu_memory_utilization=0.8,
            local_tensor_parallel_size=1,
        ),
    )
)
```

For CLI flags and operator details, refer to [Agentic retrieval](docs/cli/README.md#agentic-retrieval) in the CLI reference.

### Live RAG SDK (retrieve + answer in one call)

The pattern above -- retrieve hits, build a prompt, call an LLM -- is baked into the SDK as `Retriever.answer()` so live applications can skip the boilerplate. The same `Retriever` instance powers three entry points:

| Method | Input | Output | Use case |
| --- | --- | --- | --- |
| `Retriever.retrieve(query, top_k=...)` | one query | `RetrievalResult` (`chunks`, `metadata`) | Structured retrieval without an LLM. |
| `Retriever.answer(query, llm=..., judge=None, reference=None, ...)` | one query | `AnswerResult` (answer + chunks + optional scores) | One-shot RAG -- production/live. |
| `Retriever.pipeline().generate(...).score().judge(...).run(queries)` | many queries | `pandas.DataFrame` | Batch RAG over the operator graph, each step optional. |

Install the LLM client extra:
```bash
uv pip install "nemo-retriever[llm]"
export NVIDIA_API_KEY=nvapi-...
```

Single-query live RAG. Point `vdb_kwargs["uri"]` at any table built above; the
embedding model in `embed_kwargs` must match the one used during ingestion so
query vectors land in the same embedding space as the stored chunks.

```python
from nemo_retriever.graph.retriever import Retriever
from nemo_retriever.models.llm import LiteLLMClient

retriever = Retriever(
    vdb_kwargs={"uri": "lancedb", "table_name": "nemo-retriever"},
    embed_kwargs={
        "model_name": "nvidia/llama-nemotron-embed-1b-v2",
        "embed_model_name": "nvidia/llama-nemotron-embed-1b-v2",
        "embedding_endpoint": "https://integrate.api.nvidia.com/v1/embeddings",
    },
    top_k=5,
)
llm = LiteLLMClient.from_kwargs(
    model="nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5",
    temperature=0.0,
    max_tokens=512,
)

result = retriever.answer("What is RAG?", llm=llm)
print(result.answer)
# 'Retrieval-augmented generation combines external context with an LLM...'
print(len(result.chunks), "chunks from", {m.get("source") for m in result.metadata})
print(f"{result.latency_s:.2f}s on {result.model}")
```

Local-GPU shortcut: if you ingested with default `retriever ingest` flags
(`[local]` extra installed), drop `embed_kwargs` to reuse
the bundled `VL_EMBED_MODEL`.

Live RAG with scoring and an LLM judge (requires a ground-truth `reference`):
```python
from nemo_retriever.models.llm import LLMJudge

judge = LLMJudge.from_kwargs(
    model="nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5",
    temperature=0.1,
    max_tokens=4096,
)
result = retriever.answer(
    "What is RAG?",
    llm=llm,
    judge=judge,
    reference="RAG combines retrieved context with LLM generation.",
)
print(result.token_f1, result.judge_score, result.failure_mode)
# 0.62 4 'correct'
```

Batch RAG over the operator graph -- each builder step is optional:
```python
df = (
    retriever.pipeline()
    .generate(llm)
    .score()
    .judge(judge)
    .run(
        queries=["What is RAG?", "What is reranking?"],
        reference=["RAG combines retrieval with generation.", "Reranking re-scores retrieved passages."],
    )
)
print(df[["query", "answer", "token_f1", "judge_score", "failure_mode"]])
```

Scoring tiers on `AnswerResult`:

- **Tier 1** (`answer_in_context`) -- whether retrieval surfaced the evidence; requires `reference`.
- **Tier 2** (`token_f1`, `exact_match`) -- token-level overlap; requires `reference`.
- **Tier 3** (`judge_score`) -- dual-judge `AnswerAccuracy` LLM-as-judge score (0.0-1.0), ported from ragas onto `litellm`; requires `reference` and `judge`. `judge_reasoning` is always empty (the metric emits only a rating).
- `failure_mode` -- derived classification (`correct`, `partial`, `retrieval_miss`, `generation_miss`, `refused_*`, `thinking_truncated`).

If only `reference` is supplied, Tier 1 + 2 run. If only `judge` is supplied (without `reference`), a `ValueError` is raised. On generation error, scoring and judge are skipped and `AnswerResult.error` is populated.

### Ingest other types of content:

For PowerPoint and Docx files, ensure libeoffice is installed by your system's package manager. This is required to make their pages renderable as images for our [page-elements content classifier](https://huggingface.co/nvidia/nemotron-page-elements-v3).

For example, with apt-get on Ubuntu:
```bash
sudo apt install -y libreoffice
```

For SVG files, install the optional `cairosvg` dependency. SVG support is available in the NeMo Retriever Library, but not in the container deployment. `cairosvg` requires network access to install, so it will not work in air-gapped environments.
```bash
uv pip install "nemo-retriever[multimedia]"
# or to install only the SVG dependency:
uv pip install "cairosvg>=2.7.0"
```

Example usage:
```python
# docx and pptx files
documents = [str(Path(f"../data/*{ext}")) for ext in [".pptx", ".docx"]]
# mixed types of images
images = [str(Path(f"../data/*{ext}")) for ext in [".png", ".jpeg", ".bmp"]]
ingestor = (
  # above file types can be combined into a single job
  ingestor.files(documents + images)
  .extract()
)
```

*Note:* the `split_config` keyword on `.extract()` uses a tokenizer to split texts by a max_token length
### Render results as markdown

If you want a readable markdown view of extracted results, pass a single document's extraction
records to `nemo_retriever.common.io.to_markdown`. The helper returns one markdown string (or `None`
if there is no content), with per-page sections joined under a single document heading.

For multi-document runs, pass one document at a time—for example, `to_markdown(results[0])`.
To build a filename-keyed index across many documents, use `build_page_index`.

PDF text is split at the page level.

HTML and .txt files have no natural page delimiters, so they almost always need to be paired with the `split_config` keyword.

```python
# html and text files - include split_config to prevent texts from exceeding the embedder's max sequence length
documents = [str(Path(f"../data/*{ext}")) for ext in [".txt", ".html"]]
ingestor = (
  ingestor.files(documents)
  .extract(split_config={"text": {"max_tokens": 5}, "html": {"max_tokens": 5}}) # 1024 by default, set low here to demonstrate chunking
)
results = ingestor.ingest()
markdown_doc = to_markdown(results[0])
print(markdown_doc)
```

Use `to_markdown_by_page(results[0])` when you want a `dict[int, str]` keyed by page
number instead, where each value is the rendered markdown for that page.
For audio and video files, ensure ffmpeg is installed by your system's package manager.

For example, with apt-get on Ubuntu:
```bash
sudo apt install -y ffmpeg
```

The bundled Docker image uses the FFmpeg package provided by the base Ubuntu
image when `INSTALL_FFMPEG=true` is set. If your workflow depends on exact
FFmpeg codec or version behavior, verify the image package against those
requirements.

The bundled Dockerfile skips ffmpeg/ffprobe by default. For the service image,
set `INSTALL_FFMPEG=true` at runtime to install them during container startup:

```bash
docker run -e INSTALL_FFMPEG=true nemo-retriever-service
```

For Kubernetes deployments, set `service.installFfmpeg=true` in the Helm chart.
This runtime install requires network access to package repositories, a
writable root filesystem, and security policy that allows the image's scoped
sudo use. For locked-down environments that cannot install packages at startup,
use a custom service image that already contains ffmpeg/ffprobe.

```python
ingestor = create_ingestor(run_mode="batch")
ingestor = ingestor.files([str(INPUT_AUDIO)]).extract_audio()
```

### Store row images

Use `.store()` after `.embed()` to persist row-level image payloads to local disk or object storage (S3, MinIO, GCS via fsspec). Stored URIs are written back to the DataFrame for VDB upload and reranking.

```python
ingestor = (
  ingestor.files(documents)
  .extract()
  .embed()
  .store(
    storage_uri="s3://my-bucket/citation-assets",  # or a local path
    storage_options={"key": "...", "secret": "..."},  # fsspec auth for S3/MinIO
  )
  .vdb_upload()
)
```

### Explore Different Pipeline Options:

You can use the [Nemotron RAG VL Embedder](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2)

```python
ingestor = (
  ingestor.files(documents)
  .extract()
  .embed(
    model_name="nvidia/llama-nemotron-embed-vl-1b-v2",
    #works with plain "text"s, "image"s, and "text_image" pairs
    embed_modality="text_image"
  )
)
```

You can use a different ingestion pipeline based on [Nemotron Parse](https://build.nvidia.com/nvidia/nemotron-parse) hosted on NVIDIA Build and combined with the default embedder:

```python
ingestor = ingestor.files(documents).extract(
  method="nemotron_parse",
  nemotron_parse_invoke_url="https://integrate.api.nvidia.com/v1/chat/completions",
  nemotron_parse_model="nvidia/nemotron-parse",
)
```

## Run with remote inference, no local GPU required:

For build.nvidia.com hosted inference, set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) as an environment variable (refer to [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)).

```python
ingestor = (
  ingestor.files(documents)
  .extract(
    # for self hosted NIMs, your URLs will depend on your NIM container DNS settings
    page_elements_invoke_url="https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3",
    ocr_invoke_url="https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v2",
    table_structure_invoke_url="https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1",
  )
  .embed(
    embed_invoke_url="https://integrate.api.nvidia.com/v1/embeddings",
    model_name="nvidia/llama-nemotron-embed-1b-v2",
    embed_modality="text",
  )
  .vdb_upload()
)
```

## Ray cluster setup

NeMo Retriever Library uses Ray Data for distributed ingestion and benchmarking. [NeMo Ray run guide](https://docs.nvidia.com/nemo/run/latest/guides/ray.html)

### Local Ray cluster with dashboard

To start a Ray cluster with the dashboard on a single machine use the following command.

```bash
ray start --head
```

Open `http://127.0.0.1:8265` in your browser for the Ray Dashboard, and run your NeMo Retriever Library pipeline on the same machine with `--ray-address auto` to attach to this cluster. [Connecting to a remote Ray cluster on Kubernetes](https://discuss.ray.io/t/connecting-to-remote-ray-cluster-on-k8s/7460)

### Single‑GPU cluster on multi‑GPU nodes

To restrict Ray to a single GPU on a multi‑GPU node use the following command.

```bash
CUDA_VISIBLE_DEVICES=0 ray start --head --num-gpus=1
```
Then run your pipeline as before with `--ray-address auto` so it connects to this single‑GPU Ray cluster. [NeMo Ray run guide](https://docs.nvidia.com/nemo/run/latest/guides/ray.html)

## Multi-GPU resource heuristics (library batch mode)

### Resource heuristics (batch mode)

By default, batch mode computes resources using this order:

1. Auto-detected resources (Ray cluster if connected, otherwise local machine)
2. Environment variables
3. Explicit function arguments (highest precedence)

This means defaults are deterministic but easy to override when you need fixed behavior.

### Default behavior

- `cpu_count` / `gpu_count` are detected from Ray (`cluster_resources`) or local host.
- Worker heuristics:
  - `page_elements_workers = gpu_count * page_elements_per_gpu`
  - `detect_workers = gpu_count * ocr_per_gpu`
  - `embed_workers = gpu_count * embed_per_gpu`
  - minimum of `1` per stage
- Stage GPU defaults:
  - If `gpu_count >= 2` and `concurrent_gpu_stage_count == 3`, uses high-overlap values for page-elements/OCR/embed.
  - Otherwise uses `min(max_gpu_per_stage, gpu_count / concurrent_gpu_stage_count)`.

### Override variables

| Variable | Where to set | Meaning |
|---|---|---|
| `override_cpu_count`, `override_gpu_count` | function args | Highest-priority CPU/GPU override |

## NIM containers

For deployment of NeMo Retriever / **NIM** containers, use **Helm**:
**[`helm/README.md`](helm/README.md)** and the **NeMo Retriever Library**
documentation linked from that guide and the
[NeMo Retriever Library](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/).

## Troubleshooting

### vLLM engine fails to start during CUDA graph capture

When using the vLLM-based VL reranker, the engine may fail to start with errors similar to the following:

```
fatal error: Python.h: No such file or directory
...
torch._inductor.exc.InductorError: CalledProcessError: Command '['/usr/bin/gcc', '...cuda_utils.c', ...]' returned non-zero exit status 1.
...
RuntimeError: Engine core initialization failed.
```

This occurs because Triton compiles a small C extension at runtime during CUDA graph capture and requires the Python development headers. If `Python.h` is not installed, the compilation fails and the vLLM engine cannot start.

To resolve this, install the Python development headers for your Python version:

```bash
# For Python 3.12 on Ubuntu/Debian
sudo apt install python3.12-dev
```

After installing the headers, restart the pipeline.

## Retriever Harness

The developer harness runs registered ingest and retrieval benchmarks through
`retriever harness`. Start with the
[harness guide](harness/README.md), then choose
[library execution](harness/docs/library.md) or
[service execution](harness/docs/service.md). Recurring workstation runs use the
[nightly launcher](../ops/retriever-nightly/README.md).

### Ingest image storage

Use root ingest to persist extracted image assets to local storage or any
fsspec-compatible URI:

```bash
retriever ingest ./data \
  --store-images-uri ./processed_docs/images
```

The store stage writes the image payloads produced by ingest. With
`--embed-granularity page`, stored assets are page images. With
`--embed-granularity element`, stored assets are element images.
