Metadata-Version: 2.5
Name: mistralai-evaluations
Version: 0.7.0
Summary: Mistral AI Studio evaluation tools
Project-URL: Homepage, https://mistral.ai
Author-email: Mistral AI <support@mistral.ai>
License: Apache-2.0
License-File: LICENSE
Keywords: ai,evaluation,llm,mistral,observability
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: colorama>=0.4.0
Requires-Dist: mistralai<3.0.0,>=2.4.7
Requires-Dist: pydantic>=2.0.0
Requires-Dist: structlog>=24.0.0
Description-Content-Type: text/markdown

# Mistral Evaluations

Namespace package for Mistral AI evaluation (Dora) utilities.

## Overview

This package sets up the `mistralai.evaluations` namespace for Dora-specific utilities. It extends the Mistral AI SDK
v2 namespace structure to allow evaluation features to be developed and released from this repository.

## SDK v2 Namespace Package

This package follows [PEP 420](https://peps.python.org/pep-0420/) implicit namespace packaging to integrate with the
Mistral AI SDK v2 structure. The code lives in this repository (Dora) but will be importable from
`mistralai.evaluations` once SDK v2 is released.

### Package Structure

```
mistralai/                  # Shared namespace (no __init__.py)
├── client/                 # Core SDK (mistralai package on PyPI)
├── workflows/              # Workflows SDK
├── evaluations/            # This package (Dora utilities)
└── extra/                  # Extra utilities
```

The `mistralai/` directory has no `__init__.py`, allowing multiple packages to coexist under the shared `mistralai`
namespace.

## Installation

This package is currently under development. Once SDK v2 is released, it will be available as part of the `mistralai`
namespace.

For local development:

```bash
uv sync
```

## Usage

### Basic evaluation

```python
from mistralai.evaluations import Evaluation, Evaluator, Project

run = await client.evaluation.run(
    project=Project(name="My Project"),
    evaluation=Evaluation(name="My Evaluation"),
    dataset=[{"input": "hello", "expected": "greeting"}],
    task=lambda input_record: classify(input_record["input"]),
    evaluators=[
        Evaluator(name="accuracy", scorer=lambda input_record, output: 1 if output == input_record["expected"] else 0),
    ],
)
run.show(level="scores")
```

### Run-level evaluators

Run evaluators execute after all records are processed and receive the full run context (records, statistics, metrics, metadata). Use them for global assertions or aggregate metrics.

```python
from mistralai.evaluations import RunEvaluator

run = await client.evaluation.run(
    ...,
    evaluators=[
        Evaluator(name="accuracy", scorer=lambda input_record, output: 1 if output == input_record["expected"] else 0),
    ],
    run_evaluators=[
        RunEvaluator(
            name="accuracy_above_50pct",
            scorer=lambda ctx: ctx.statistics["accuracy"].avg > 0.5,
        ),
        RunEvaluator(
            name="all_records_scored",
            scorer=lambda ctx: all(
                len(r.output.generations) > 0 for r in ctx.records
            ),
        ),
    ],
)
# Results are in run.run_scores
print(run.run_scores)  # {"accuracy_above_50pct": True, "all_records_scored": True}
```

The `RunEvaluatorContext` provides:

| Field        | Type                             | Description                                   |
| ------------ | -------------------------------- | --------------------------------------------- |
| `records`    | `list[RunEvaluatorRecord]`       | Each record's input and output (with scores). |
| `statistics` | `dict[str, EvaluatorStatistics]` | Run-level statistics from regular evaluators. |
| `metrics`    | `dict[str, JsonValue]`           | Run-level `run_aggregator` results.           |
| `metadata`   | `dict[str, JsonValue]`           | Metadata attached to the run.                 |

### Retrying failed records

If an evaluation only partially fails, use `retry_failed_records` instead of re-running the entire evaluation from
scratch. It re-runs only the failed records and patches the original run in place, avoiding work on successful records
when failures came from transient API errors, rate limits, or a bug in the task or scorer.

```python
from mistralai.evaluations import Evaluation, Evaluator

run = await client.evaluation.run(
    evaluation=Evaluation(name="My Evaluation"),
    dataset=dataset,
    task=flaky_task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
)

# Fix the task or scorer if needed, then retry only the failed records.
result = await client.evaluation.retry_failed_records(
    run_id=run.run_id,
    dataset=dataset,
    task=fixed_task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
)

print(f"Retried: {result.retried_count}, Patched: {result.patched_count}")
```

Pass the same `dataset` used in the original run so the SDK can map failed records back to their inputs. You can pass a
fixed `task` or updated evaluators before retrying. When run-level evaluators are provided, their scores are recomputed
after patching.

### `get_score` helper

When writing run evaluators, accessing individual scores requires navigating through generations and score lists. The `get_score` helper simplifies this:

```python
from mistralai.evaluations import RunEvaluatorContext, Score, get_score

def f1_scorer(ctx: RunEvaluatorContext) -> Score:
    tp = fp = fn = 0
    for record in ctx.records:
        expected = str(record.input["expected"]).lower()
        # Instead of: record.output.generations[0].scores["accuracy"][0].value
        is_correct = get_score(record, "accuracy").value == 1
        ...
```

Returns a `Score` object (with `value`, `rationale`, `metadata`).

| Parameter        | Type                             | Default | Description                                                 |
| ---------------- | -------------------------------- | ------- | ----------------------------------------------------------- |
| `record`         | `RunEvaluatorRecord`             | —       | The record to extract scores from.                          |
| `evaluator_name` | `str`                            | —       | Name of the evaluator whose scores to retrieve.             |
| `aggregate`      | `Callable[[list[Score]], Score]` | `None`  | Required when multiple scores exist (multiple generations). |

Raises if no scores are found. When `num_generations > 1`, raises with a helpful message unless an `aggregate` function is provided. For pre-computed aggregations, use `record.output.statistics[evaluator_name]` instead.

## Development

```bash
# Install dependencies
uv sync

# Run tests
uv run pytest

# Lint and format
uv run ruff check --fix .
uv run ruff format .

# Type check
uv run mypy mistralai
```

## Publishing

### Release from GitHub

The version is **not** stored in code. This package uses
[`uv-dynamic-versioning`](https://github.com/ninoseki/uv-dynamic-versioning): the published version comes from the
release tag (`evaluations-sdk/vX.Y.Z`), and the workflow stamps the version you pass onto the build via
`UV_DYNAMIC_VERSIONING_BYPASS`. **There is no version-bump PR** — just run the workflow with the version you want.

Run the "Release Evaluations SDK" GitHub workflow manually. Choose which registries to publish to via the boolean
inputs — they're independent, so you can dogfood internally without shipping to clients:

1. **PyPI** (public), via a trusted publisher (OIDC) — the primary channel. Default on.
2. **Gemfury**, for internal distribution. Default on.
3. **Cloudsmith** (`mistral-ai/sdk-distribution`), for private-preview client distribution. Default off — opt in when previewing to specific clients.

Run the workflow from `main` for normal SDK releases, or from the relevant `mais-*` release branch for on-prem patch
lines. Enter the version as the workflow `version` input (PEP 440, e.g. `0.7.0` or `0.7.0rc1`); the workflow fails if
the tag `evaluations-sdk/vX.Y.Z` already exists.

The git tag `evaluations-sdk/vX.Y.Z` records the released commit for that version, so it is created as soon as **any**
selected registry publishes successfully (and none of the selected ones failed). To dogfood privately, publish an `rc`
to Gemfury only; the rc tag won't collide with the later public release.

If one registry succeeds and another fails, rerun the failed GitHub Actions jobs only. Do not rerun the full workflow,
because the already-successful registry may reject the duplicate version.

The TypeScript mirror (`ts/packages/local-observability-sdk/`) still carries its own `package.json` / `cli.ts` version
and is not published by this workflow; keep it in sync when practical.

### Manual Gemfury publish

Retrieve the "Gemfury Upload Token - Mistral" credentials from Bitwarden. The `make publish` command expects `GEMFURY_USERNAME` and `GEMFURY_PASSWORD` environment variables.

```bash
# Publish to Gemfury
# this will clean up the dist/ directory then build and publish
make publish
```

### Installing from PyPI (public)

```bash
pip install mistralai-evaluations
```

### Installing from Gemfury (internal)

```bash
pip install --index-url https://pypi.fury.io/mistralai/ mistralai-evaluations --extra-index-url https://pypi.org/simple/
```
