Metadata-Version: 2.4
Name: arara-rag
Version: 0.5.1
Summary: Portuguese-first, CPU-only retrieval: tinyzchunk chunking, static embeddings, BM25, rank fusion and CXM25 reranking. numpy only, out-of-core, with metadata filtering and CRUD.
Author: Carlo Moro
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/cnmoro/Arara-RAG
Project-URL: Repository, https://github.com/cnmoro/Arara-RAG
Keywords: rag,retrieval,portuguese,pt-br,embeddings,bm25,cpu,numpy,out-of-core
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Indexing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: tokenizers>=0.20
Requires-Dist: model2vec>=0.9
Requires-Dist: tinyzchunk>=0.3.3
Requires-Dist: cxm25>=0.1.0
Requires-Dist: threadpoolctl>=3.1
Provides-Extra: nanoe5
Requires-Dist: nanoe5>=0.3; extra == "nanoe5"
Provides-Extra: bench
Requires-Dist: datasets>=2.19; extra == "bench"
Requires-Dist: pyarrow; extra == "bench"
Requires-Dist: huggingface_hub>=0.23; extra == "bench"
Requires-Dist: matplotlib>=3.8; extra == "bench"
Provides-Extra: validate
Requires-Dist: pytrec_eval-terrier>=0.5.6; extra == "validate"
Requires-Dist: scikit-learn>=1.3; extra == "validate"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# arara-rag

**Portuguese-first retrieval that runs entirely on CPU.** Chunking, dense and
lexical retrieval, rank fusion and reranking — numpy only. No PyTorch, no ONNX
Runtime, no FAISS. The whole install is **200 MB**.

```bash
pip install arara-rag
```

```python
from arara_rag import Arara

arara = Arara(path="./indice")            # out-of-core and persistent
arara.add_documents(
    {"lei_1234": open("lei.txt").read()},
    metadata={"ano": 2024, "tipo": "lei", "uf": "BR"},
)
hits = arara.search("qual a alíquota?", top_k=5, where={"ano": {"$gte": 2020}})
print(hits[0].doc_id, arara.resolve(hits[0]))     # exact source span
```

## What's in it

| Stage | Component | Size |
|---|---|---|
| Chunking | [`tinyzchunk`](https://github.com/cnmoro/tinyzchunk) — tokenizer-free, distilled from an LLM teacher | 2.1 MB |
| Dense | [`static-nomic-384-pten-v2`](https://huggingface.co/cnmoro/static-nomic-384-pten-v2) — Model2Vec static embeddings | 62 MB |
| Lexical | BM25 over a numpy inverted index | — |
| Rerank | [`CXM25`](https://github.com/cnmoro/CXM25) — PT-BR lexical scoring | bundled |
| Fusion | Reciprocal Rank Fusion | — |

An index can be **out-of-core**: vectors live in a memory-mapped file and
documents, metadata and offsets in SQLite, so the index is about **1 KB per
document** and cold pages can be evicted by the OS instead of being pinned on
the heap. Same API either way — `Arara()` keeps everything in memory.

## Speed and memory

One CPU core, no GPU. Measured end to end with `python -m bench.profile`.

| documents | index build | query p50 | query p95 | index size | peak RSS to serve |
|---|---|---|---|---|---|
| 1,000 | 3.1 s | **0.45 ms** | 0.54 ms | 1.8 MB | 472 MB |
| 10,000 | 8.8 s | **0.86 ms** | 6.8 ms | 18 MB | 481 MB |
| 50,000 | 33.8 s | **7.0 ms** | 8.0 ms | 91 MB | 553 MB |

- **~1,300–1,500 documents/second** to chunk, embed, tokenise and index —
  chunking and embedding are per-document, so they run across processes.
  A single process manages ~300/second.
- **~1.8 KB per document** of index: 1.5 KB of vectors plus BM25 postings.
- Query latency scales with corpus size because both retrievers score the whole
  corpus per query — that is what makes the ranking exact rather than
  approximate.
- Peak RSS is dominated by a **~470 MB fixed cost** (Python, numpy, the
  embedding model and the tokenizer tables); the corpus adds ~1.7 KB per
  document on top. Vectors are memory-mapped, so cold pages can be evicted.

![Speed and memory](docs/scaling.png)

## Retrieval quality

MTEB-BR, the Brazilian Portuguese benchmark with a
[public leaderboard](https://huggingface.co/spaces/MTEB-BR/leaderboard).
nDCG@10, fixed-window chunking. Metrics are computed by `bench/metrics.py`,
which `bench/validate_metrics.py` checks against `pytrec_eval` to **0.0e+00**,
and against scikit-learn on binary relevance.

Both dense backends are shown, because the choice matters more than any
parameter in the stack. `static` is the default; `nanoE5` is opt-in (see below).

| Task | docs | rel./query | static dense | nanoE5 dense | static hybrid | nanoE5 hybrid | lexical |
|---|---|---|---|---|---|---|---|
| BRTaxQAR (capped) | 478 | 2.92 | 0.2934 | 0.3423 | 0.3486 | **0.4180** | 0.4051 |
| FaQuADIR | 244 | 1.0 | 0.7139 | 0.8314 | 0.8304 | 0.8906 | **0.8961** |
| FaqBacenRetrieval | 1,673 | 1.0 | 0.3744 | 0.5858 | 0.4526 | 0.5659 | 0.4881 |
| JurisTCU | 16,045 | 15.0 | 0.3887 | 0.4906 | 0.4890 | **0.5685** | 0.5378 |
| Quati | 50,000 | 38.66 | 0.3268 | _not run_\* | 0.4046 | _not run_\* | 0.4067 |

\* Quati's nanoE5 index did not finish in the time available — it is feasible
(~15 minutes of encoding), but this host was carrying load from other tenants
and three attempts stalled past 45 minutes. Everything else in this table was run
twice with identical results.

The last column is BM25 alone. Note what changes with a real encoder: **hybrid
beats lexical on three of the four rows where both were measured**, and ties it
on FaQuADIR. With the static model lexical won every row, so the argument for
hybrid retrieval was not visible in these numbers until nanoE5 was added.

![Dense and hybrid per backend against BM25](docs/encoders.png)

**Read `rel./query` before comparing across rows.** nDCG@10 measures very
different things on these tasks, and that is a property of the benchmarks rather
than of the retrieval:

- **FaQuADIR and FaqBacen have exactly one relevant document per query.** There
  nDCG@10 is a transform of the rank of that one document, so 0.90 means it is
  usually first and 1.0 is the ceiling.
- **Quati has 38.66 relevant documents per query.** With ten slots, a perfect
  score requires all ten to be relevant, so 0.41 means roughly four of the top
  ten are relevant — much closer to precision at 10 than the FaQuADIR number is.
- JurisTCU (15.0) and BRTaxQAR (2.92) sit in between.

A retriever scoring 0.33 on Quati is therefore not "worse" than one scoring 0.71
on FaQuADIR — the numbers are not comparable across rows, only within one.

### Choosing a dense encoder

Two backends ship. The default is a Model2Vec lookup table; [nanoE5.c](https://github.com/cnmoro/nanoe5.c)
is an opt-in alternative — a 4-bit `multilingual-e5-small` in C, with no PyTorch,
no ONNX and no BLAS. It is a real transformer forward pass, so it is slower, and
it is a separate dependency:

```bash
pip install "arara-rag[nanoe5]"
```
```python
Arara(dense_backend="nanoe5")                              # or "static", the default
Arara(dense_backend="nanoe5", dense_variant="original")    # English-first build
```

It is meaningfully better: **dense retrieval improves by 0.05–0.21 nDCG@10** —
on FaqBacen that is a larger jump than the whole distance from BM25 to the
leaderboard median. The comparison table above is the full picture.

The cost is indexing speed, and it tracks chunk length because nanoE5 windows
anything past 512 tokens. On a fixed corpus of 300 passages of ~1.1 kB:

| | static | nanoE5 |
|---|---|---|
| index build | 2.5 s (121 docs/s) | 14.4 s (21 docs/s) |
| query p50 | 0.5 ms | 13 ms |

Longer chunks cost more than proportionally — a corpus of 2.5 kB chunks is far
slower again — while queries are one short sequence either way. nanoE5's query
latency also has a longer tail than the static model's, so prefer `static` where
p99 matters more than recall.

CXM25 reranking on top of the hybrid adds **+0.018 to +0.077 nDCG@10** across
these tasks for 1–6 ms per query (FaQuADIR: 0.8304 → **0.9078**,
BRTaxQAR full documents: 0.4801 → **0.5091**).

## Why chunking matters most

Legal documents in BR-TaxQA-R average 32,000 characters and reach 1.17M. MTEB-BR
truncates them at 32k because transformer encoders cannot fit more. arara
chunks, so it indexes the whole statute.

| Configuration | chunks | nDCG@10 | R@100 |
|---|---|---|---|
| capped at 32k, one vector per doc *(the leaderboard's setting)* | 478 | 0.1496 | 0.4351 |
| capped at 32k, fixed windows | 2,552 | 0.2934 | 0.6300 |
| capped at 32k, **tinyzchunk** | 23,319 | 0.3088 | 0.6225 |
| **full documents**, fixed windows | 6,439 | 0.4041 | 0.7497 |
| **full documents**, paragraph splits | 6,439 | 0.4041 | 0.7497 |
| **full documents**, tinyzchunk | 60,927 | 0.4287 | 0.7486 |
| **full documents**, tinyzchunk + BM25 | 60,927 | 0.4801 | 0.8496 |
| **full documents**, + CXM25 rerank | 60,927 | **0.5091** | 0.8102 |

![Chunking a legal corpus beats truncating it by 3.4×](docs/ablation.png)

This ablation is measured with the **static** backend. nanoE5 is not run here,
and the reason is a property of that model rather than a gap in the table: it
windows anything past 512 tokens, so BR-TaxQA's 2.5 kB chunks cost it about
0.9 chunks/second. The capped corpus alone would take ~17 minutes and the
full-document, tinyzchunk configuration — 60,927 such chunks — would take most
of a day. The comparison that matters for nanoE5 is in
[Retrieval quality](#retrieval-quality), where a real encoder changes the
conclusion about hybrid retrieval.

## Against the leaderboard

Best configuration per backend, and how much of the field each beats:

| Task | static best | nanoE5 best | best on the leaderboard | leaderboard median |
|---|---|---|---|---|
| FaQuADIR | **0.9078** (hybrid+CXM25) | 0.8906 (hybrid) | voyage-context-4 (0.8738) | 0.7689 |
| BRTaxQAR | 0.4051 (lexical) | **0.4180** (hybrid) | voyage-finance-2 (0.4499) | 0.2723 |
| JurisTCU | 0.5378 (lexical) | **0.5685** (hybrid) | llama-embed-nemotron-8b (0.6805) | 0.5436 |
| FaqBacenRetrieval | 0.4881 (lexical) | **0.5858** (dense) | codestral-embed (0.8262) | 0.6516 |
| Quati | 0.4211 (hybrid+CXM25) | _not run_ | voyage-context-4 (0.6901) | 0.5569 |

| Task | static dense | static hybrid | static lexical | nanoE5 dense | nanoE5 hybrid |
|---|---|---|---|---|---|
| FaQuADIR | 31% | 80% | **100%** | 81% | **100%** |
| BRTaxQAR | 56% | 76% | 95% | 71% | **97%** |
| JurisTCU | 25% | 34% | 47% | 35% | **62%** |
| FaqBacenRetrieval | 18% | 27% | 30% | **33%** | 30% |
| Quati | 22% | 29% | 29% | - | - |

Percentages are the share of the 95-96 published models each beats.
**On FaQuADIR arara outranks every model on the board**, and on BRTaxQAR the
nanoE5 hybrid beats 92 of 95. The static model's dense retrieval is the weak cell
in every row — 18% on FaqBacen, 25% on JurisTCU — and nanoE5 lifts both.

The honest caveat: the leaderboard evaluates *embedding* models, and there is no
BM25 entry on it. arara's strongest static modes are lexical, and lexical
retrieval is simply very good on short, high-overlap PT-BR documents — part of
that gap is a missing baseline on their side.

![arara against the MTEB-BR leaderboard](docs/leaderboard.png)

## Reranking

MTEB-BR reranking hands you a fixed candidate list and scores only the order
(MAP@1000), so `identity` is the baseline the benchmark ships with.

| Task | identity | static dense | nanoE5 dense | static hybrid | nanoE5 hybrid | CXM25 |
|---|---|---|---|---|---|---|
| QuatiReranking | 0.2839 | 0.2798 | **0.5003** | 0.3066 | 0.4300 | 0.3100 |
| JurisTCUReranking | 0.4150 | 0.3609 | 0.4616 | 0.4129 | 0.4698 | **0.4845** |

This is the sharpest version of the same story. On QuatiReranking the static
dense model *regresses* on the candidate order it was given (0.2839 → 0.2798),
while nanoE5 dense improves it by **+0.22 MAP@1000** and beats every static mode,
CXM25 included. Lexical reranking is unaffected by the backend, as it must be.

## Out-of-core, metadata, CRUD

An index is read far more than it is written, so deletes are tombstones and
freed slots are recycled on the next write.

```python
arara = Arara(path="./indice", max_chunk_chars=2000)

arara.add_documents(docs, metadata={"ano": 2024})        # insert / replace
arara.update_metadata("lei_1234", {"revisado": True})    # no re-embedding
arara.delete_document("lei_1234")                        # tombstone + slot reuse
arara.get_document("lei_1234")                           # (text, metadata)
arara.compact()                                          # reclaim file space

arara.search(q, where={"tipo": {"$in": ["lei", "decreto"]}, "ano": {"$gte": 2020}})
arara.search(q, where={"$or": [{"uf": "SP"}, {"uf": "RJ"}]})
```

Supported per field: `$eq` (bare value), `$ne`, `$gt`, `$gte`, `$lt`, `$lte`,
`$in`, `$nin`, `$exists`, `$contains`, `$startswith`, `$endswith`; plus
top-level `$and` / `$or`. Field names are validated and values are bound as SQL
parameters, so a filter cannot inject SQL.

## Guarantees

Enforced by 88 tests, not asserted in prose:

- every chunk is an **exact substring** of the canonical document, ordered and
  non-overlapping, with only whitespace between chunks — nothing is dropped;
- **no chunk exceeds `max_chunk_chars`**, including on a 24,000-character line;
- CRLF and LF inputs chunk **identically** and offsets still resolve;
- the in-memory and on-disk paths return **identical rankings**;
- importing the package never imports `torch` or `onnxruntime`.

## Reproduce

```bash
python -m venv .venv && .venv/bin/pip install -e ".[bench,validate,dev]"
python -m pytest tests/                 # 88 tests
python bench/validate_metrics.py        # metrics vs pytrec_eval
./bench/run_all.sh                      # every suite -> bench/results/
python -m bench.profile                 # speed and memory -> docs/scaling.png
python -m bench.charts                  # regenerate the figures
python -m bench.leaderboard             # compare against MTEB-BR
```

## Layout

```
arara_rag/
  chunk.py      chunking and the losslessness contract
  dense.py      static encoder
  lexical.py    BM25 inverted index + CXM25 reranker
  store.py      memory-mapped vectors, SQLite catalog, filters
  pipeline.py   Arara: add / search / rerank / CRUD
bench/          task loaders, metrics, suites, profiling, charts
tests/          88 contract and correctness tests
space/          Gradio demo
```

## Limitations

- **PT-BR and English.** The tokenizer, stemmer and stopwords are Portuguese.
- **The dense model is small** and static; lexical retrieval carries the stack
  on short, high-overlap documents.
- **Per-chunk bookkeeping stays resident** (16 bytes/chunk); vectors, text and
  metadata do not.
- **CXM25 reranking is ~71 µs/document**, so it runs over a candidate set.
- **Index build forks worker processes** to parallelise chunking and embedding.
  Set `workers=1` where forking is unsafe or unavailable; the result is
  byte-identical, only slower.

## License

Apache-2.0.
