Metadata-Version: 2.4
Name: dj-evals
Version: 0.7.0
Summary: Internal Django SSE eval runner
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: django>=5.1
Requires-Dist: openai>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: typing-extensions>=4.12
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Provides-Extra: dev
Requires-Dist: ruff>=0.8; extra == "dev"

# dj-evals

Internal Django helper for running one eval function with multiple argument sets in parallel, so you can visually compare output, tool calls, usage, cost, and completion behavior as the runs happen.

## Django setup

Add `dj_evals` to `INSTALLED_APPS`:

```python
INSTALLED_APPS = [
    "dj_evals",
]
```

Create one view that connects the package to your project. This is the only view required by `dj-evals`:

```python
from django.contrib.admin.views.decorators import staff_member_required
from dj_evals import handle_eval_request


@staff_member_required
async def eval_run(request):
    return await handle_eval_request(
        request,
        allowed_paths={"myapp.evals.echo_eval"},
    )
```

Mount that view in your URL configuration:

```python
from django.urls import path

urlpatterns = [
    path("evals/run/", eval_run, name="eval-run"),
]
```

Replace `staff_member_required` with your project's authorization check if staff-only access is not appropriate.

## Link to eval suites

A project can have a small page that lists its eval suites. Use `generate_eval_url()` to build the URL behind each link:

```python
from django.contrib.admin.views.decorators import staff_member_required
from django.shortcuts import render
from django.urls import reverse
from dj_evals import generate_eval_url


@staff_member_required
def eval_index(request):
    question = (
        "Give me 7 tips to improve my game in chess. "
        "My ELO is 500. I make lots of blunders."
    )
    url = reverse("eval-run") + generate_eval_url(
        "myapp.evals.echo_eval",
        {
            "model": "gpt-5.6-terra",
            "question": question,
        },
        {
            "model": "gpt-5.6-luna",
            "question": question,
        },
        # Optional. Requires OPENAI_API_KEY.
        expectations=[
            "Don't leave things hanging",
            "Castle early",
            "The answer mentions using a checklist",
            "The checklist is something simple yet effective",
        ],
    )
    return render(
        request,
        "eval_index.html",
        {"eval_suites": [("Chess tips for beginners", url)]},
    )
```

Render those URLs as normal links in `eval_index.html`:

```django
{% for name, url in eval_suites %}
  <a href="{{ url }}">{{ name }}</a>
{% endfor %}
```

Clicking the link opens the comparison page. It starts one POST request per argument set and renders the streamed results side by side.

Expectations and **Auto-rate** use `OPENAI_API_KEY` to judge and compare completed runs.

## Eval function contract

The eval function must be importable by dotted path and whitelisted in `allowed_paths`. It can be a sync or async function that returns one event, or an async generator that yields events.

```python
def echo_eval(question="hello", model=""):
    return question


async def async_eval(question="hello", model=""):
    return question


async def streaming_eval(question="hello"):
    yield {"type": "response.output_text.delta", "delta": question}
    yield {"type": "response.output_text.done", "text": question}
    yield {"type": "response.completed", "response": {"cost": 0}}
```

Eval argument dictionaries are passed to the eval function as keyword arguments.

## Helper utilities

You can yield `EvalEvent` for custom progress messages in the output panel:

```python
from dj_evals import EvalEvent


async def streaming_eval():
    event: EvalEvent = {
        "type": "dj_evals.event",
        "message": "Fetched documents",
    }
    yield event
```

## Developers

Run a local demo server to try the library in a browser:

```bash
uv run --with daphne daphne -p 8003 examples.local_server:application
# or, if you have just installed:
just run
```

Then open <http://127.0.0.1:8003/> and choose an example. The OpenAI example requires `OPENAI_API_KEY`.

Run checks from this repo:

```bash
uv run --with ruff ruff check .
uv run --with pytest --with pytest-asyncio pytest -q
# or, if you have just installed:
just test
```
