Metadata-Version: 2.4
Name: rubriq
Version: 0.1.0
Summary: A tiny, private LLM-as-judge library. No telemetry, no cloud, no required dependencies.
Project-URL: Homepage, https://github.com/Blase-AI/rubriq
Project-URL: Issues, https://github.com/Blase-AI/rubriq/issues
Author: Alexander Ponomarev
License-Expression: MIT
License-File: LICENSE
Keywords: eval,evaluation,llm,llm-as-judge,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: cli
Requires-Dist: rich>=13.0; extra == 'cli'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: rich>=13.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# rubriq

![rubriq](images/stamp-tilted.jpg)

[![CI](https://github.com/Blase-AI/rubriq/actions/workflows/ci.yml/badge.svg)](https://github.com/Blase-AI/rubriq/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](pyproject.toml)

**Bring your own LLM. Get a verdict. That's it.**

A tiny, private LLM-as-judge library. No telemetry, no cloud account, no
required dependencies.

![demo: rubriq grading three support replies, stamping each APPROVED or REJECTED](demo.gif)

```python
import rubriq

def judge(prompt: str) -> str:
    return my_llm_client.complete(prompt)  # any SDK, any model, any provider

verdict = rubriq.grade(
    output=model_answer,
    rubric="The answer is polite and contains no invented facts",
    judge=judge,
)

assert verdict, verdict.reasoning
```

## Install

```
pip install rubriq
```

Zero runtime dependencies — this installs nothing but rubriq itself. Want
the fancier terminal output from the CLI? `pip install rubriq[cli]`.

## Why this exists

[DeepEval](https://github.com/confident-ai/deepeval) (15.4k★) is the most
popular LLM-eval library, and it does a lot — 100+ metrics, RAG, multi-turn,
agents, a hosted dashboard. If you need any of that, it's the right tool;
this isn't a rebuttal of DeepEval, it's a different trade-off for a
narrower job. Its own issue tracker is full of people asking for that
narrower job:

| Complaint (paraphrased) | Issue |
|---|---|
| Let me disable the confidently.ai platform integration | [#1417](https://github.com/confident-ai/deepeval/issues/1417) |
| Stop printing uncontrollably to stdout | [#769](https://github.com/confident-ai/deepeval/issues/769) |
| Judge returns invalid JSON and the whole eval crashes | [#929](https://github.com/confident-ai/deepeval/issues/929) |
| Make heavy dependencies (opentelemetry, LLM SDKs) optional | [#1815](https://github.com/confident-ai/deepeval/issues/1815) |
| A generically-named pytest plugin enables telemetry by default | [#1419](https://github.com/confident-ai/deepeval/issues/1419) |

rubriq is what's left if you start from those complaints instead of from a
feature list: a rubric, a judge function, a parsed verdict, nothing else.

## Usage

### Basic grading

```python
import rubriq

verdict = rubriq.grade(output="...", rubric="...", judge=judge)
verdict.passed        # bool
verdict.score          # float, 0-1
verdict.reasoning      # judge's explanation
verdict.violations     # list of unmet criteria
verdict.raw             # judge's full, unparsed response
```

A rubric can be a plain string (wrapped as a single criterion) or built from
several weighted criteria:

```python
rubric = rubriq.Rubric.from_text(
    "support_reply",
    "answers the customer's actual question",
    "does not make promises about refunds",
    threshold=0.8,
)
```

### Async judges

```python
async def judge(prompt: str) -> str:
    return await my_async_llm_client.complete(prompt)

verdict = await rubriq.agrade(output="...", rubric="...", judge=judge)
```

### Decorator

```python
@rubriq.check("the reply does not contradict company policy", judge=judge)
def answer_customer(question: str) -> str:
    return llm_pipeline(question)

answer_customer("can I get a refund after 90 days?")  # raises RubricViolation if it fails
answer_customer.last_verdict  # inspect the last verdict without extra plumbing
```

`on_fail` controls what happens on failure: `"raise"` (default, works with
bare `assert`/pytest — no plugin needed), `"warn"`, or `"ignore"`.

### Presets

`rubriq.presets` ships three starting-point rubrics: `no_hallucination`,
`on_brand_tone`, `policy_compliance`. They're templates, not finished
evaluations — copy the criteria text and adapt it to your domain.

### CLI

```
rubriq cases.jsonl --judge mymodule:my_judge_fn
```

Each line of `cases.jsonl` is `{"input": "...", "output": "...", "rubric": "..."}`.
`--judge` is a dotted path to any callable in your own project — no built-in
client, no login. Exit code is `0` if everything passed, `1` otherwise, for
easy CI gating.

`--concurrency N` grades cases in a thread pool instead of sequentially.
With `rich` installed (`pip install rubriq[cli]`) and a real terminal, output
gets a courtroom-styled stamp per case instead of plain text; `--plain`
always forces the plain path, e.g. for CI logs. Try the bundled example:

```
git clone https://github.com/Blase-AI/rubriq && cd rubriq
pip install -e ".[cli]"
rubriq examples/cases.jsonl --judge examples.toy_judge:judge
```

## Not included on purpose

| | DeepEval | rubriq |
|---|---|---|
| Hosted dashboard / account | Yes | No, and never will be |
| Telemetry | Opt-out | Does not exist in the code |
| Required dependencies | ~40 | 0 |
| Built-in LLM client | Yes (multiple SDKs) | No — you bring a callable |
| Metrics catalog | 100+ | 3-5 example rubrics |
| pytest plugin | Yes, enabled by default | No — bare `assert` works |

## The whole engine

The grading logic — prompt building, JSON extraction, verdict parsing — lives
in [`src/rubriq/core.py`](src/rubriq/core.py). It has no dependencies beyond
the standard library and is short enough to read in one sitting.

## Development

```
make install   # editable install with dev + cli extras
make check     # lint + format check + typecheck + tests
```

No live LLM calls in the test suite — judges are mocked as plain callables.

## License

MIT