Metadata-Version: 2.4
Name: nufi-python
Version: 1.1.1
Summary: Python SDK and CLI for the NuFi platform.
Author: Dudaji Inc.
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.25.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: tuspy==1.1.0
Requires-Dist: typer>=0.16.0
Requires-Dist: rich>=13.0.0
Provides-Extra: evaluator
Requires-Dist: pyarrow>=24.0.0; extra == "evaluator"
Dynamic: license-file

# NuFi Python SDK/CLI

`nufi-python` is the Python SDK and CLI for a NuFi installation. It requires Python
3.11 or newer and installs the `nufi` Python package and `nufi` command.

## Installation

```bash
python -m pip install nufi-python
nufi --version
nufi --help
```

Install the optional evaluator dependencies when writing a custom evaluator that reads
NuFi Parquet datasets:

```bash
python -m pip install "nufi-python[evaluator]"
```

## Configure access

Ask your NuFi administrator for the installation's base domain and a project you can
access. Create a profile, log in, and select the project once:

```bash
nufi profile create production --base-domain nufi.example
nufi profile use production
nufi login
nufi project use team-a
```

`login` uses the device flow. Add `--no-browser` on a headless host and open the
printed verification URL elsewhere.

## CLI quickstart

The CLI workflow is composable; every long-running identity can be resumed with its
dedicated `wait` command:

```bash
nufi model version create qwen2 v1 --volume models --path /qwen2
nufi compile create qwen2 --version v1 --wait
nufi compile wait COMPILE_NAME_OR_ID
nufi dataset list
nufi evaluation run create METHOD_NAME --model qwen2 \
  --model-version v1 --artifact original --wait
nufi evaluation run wait RUN_ID
nufi leaderboard create quality --method METHOD_NAME \
  --rank-metric exact_match --rank-direction desc
nufi leaderboard submission create quality --model qwen2 \
  --model-version v1 --artifact original --wait
nufi leaderboard submission wait quality SUBMISSION_ID
nufi leaderboard entries quality
```

## Python SDK quickstart

The same flow is available in Python:

```python
from nufi import NuFi

with NuFi.from_profile() as client:
    compile_run = client.compiles.create(model_name="qwen2", version="v1")
    compiled = client.compiles.wait(compile_run.id)

    evaluation_run = client.evaluations.create(
        evaluation_method_id="METHOD_ID",
        model_artifact_id=9,
        cpu="4",
        memory="8Gi",
        accelerator_count=1,
    )
    evaluated = client.evaluations.wait(evaluation_run.id)

    leaderboard = client.leaderboards.create(
        "quality",
        evaluation_method_id="METHOD_ID",
        ranking_policy={
            "primary": {
                "metric": {"source": "aggregateMetrics", "name": "exact_match"},
                "direction": "desc",
            }
        },
    )
    submission = client.leaderboards.submit(
        leaderboard.id,
        model_artifact_id=9,
    )
    completed = client.leaderboards.wait(leaderboard.id, submission.id)
    entries = client.leaderboards.entries(leaderboard.id)
```

## Evaluations

Evaluation Method metadata is unconditional and v2-only. Method config is owned and
validated by the API; the SDK forwards the mapping and any caller-supplied Method type
without a client-side allowlist.

The `nufi.evaluator.run` helper exits non-zero when evaluation or result submission
fails. NuFi determines Run success from Job success and durable API acceptance;
evaluator output is diagnostic only.

`evaluation method templates` currently exposes lm-eval templates, metrics, and
filters. It is an optional discovery command and is not needed when you already have a
Method config.

```python
from nufi import NuFi

client = NuFi.from_profile()
project = "team-a"
config = {
    "schemaVersion": "nufi.evaluation.method.v2",
    "runnerType": "lm_eval",
    "template": "short_answer",
    "datasetMapping": {"input": "question", "reference": "answer"},
    "nativeConfig": {
        "output_type": "generate_until",
        "doc_to_text": "{{input}}",
        "doc_to_target": "{{reference}}",
        "metric_list": [{"metric": "exact_match"}],
    },
}

templates = client.evaluations.methods.templates(project=project)
preview = client.evaluations.methods.preview(
    "DATASET_ID", "VERSION_ID", project=project, config=config
)
method = client.evaluations.methods.create(
    "DATASET_ID", "VERSION_ID", "gsm8k-exact-match", project=project, config=config
)
run = client.evaluations.create(
    project=project,
    evaluation_method_id=method.id,
    model_artifact_id=9,
    cpu="1",
    memory="2Gi",
    accelerator_count=1,
)
result = client.evaluations.get(run.id, project=project)
client.evaluations.cancel(run.id, project=project)
client.close()
```

The CLI accepts Dataset, Dataset Version, and Evaluation Method names or UUIDs. Run
arguments remain UUIDs because Runs have no name. Select a Model Artifact with
`--model`, `--model-version`, and `--artifact`; `--model-artifact-id` remains available
for automation. Config, default-runtime, and runner-config files may be JSON or YAML
and must have a mapping at the document root.

```yaml
# method.yaml
schemaVersion: nufi.evaluation.method.v2
runnerType: lm_eval
template: short_answer
datasetMapping:
  input: question
  reference: answer
nativeConfig:
  output_type: generate_until
  doc_to_text: "{{input}}"
  doc_to_target: "{{reference}}"
  metric_list:
    - metric: exact_match
```

```bash
nufi evaluation method templates --project team-a
nufi evaluation method validate gsm8k v1 --config method.yaml --project team-a
nufi evaluation method create gsm8k v1 gsm8k-exact-match \
  --config method.yaml --project team-a
nufi evaluation method list --dataset gsm8k --version v1 --project team-a
nufi evaluation method archive gsm8k-exact-match --yes --project team-a
nufi evaluation method restore gsm8k-exact-match --project team-a

nufi evaluation run create gsm8k-exact-match \
  --model qwen2 --model-version v1 --artifact original \
  --project team-a
nufi evaluation run list --dataset gsm8k --version v1 --project team-a
nufi evaluation run get RUN_ID --project team-a
nufi evaluation run cancel RUN_ID --yes --project team-a
```

Run creation defaults to `--cpu 4`, `--memory 8Gi`, and `--accelerator-count 1`;
pass any option to override its default.

Run detail table output shows aggregate and diagnostic summaries only. Use explicit
`--output json` to include the authorized bounded diagnostic input, output, reference,
score, and error fields returned by the API. Additive response fields are preserved;
legacy Method/Run responses missing required v2 fields are rejected.
