Metadata-Version: 2.5
Name: phigrade
Version: 3.4.0
Summary: A lightweight autograder where tests run locally and data is stored remotely
Project-URL: Repository, https://github.com/mld-instructors/phigrade
Author-email: Matt Gormley <mgormley@cs.cmu.edu>, Jacob Rast <jrast@andrew.cmu.edu>
License-Expression: MIT
License-File: LICENSE
Keywords: autograder,education,grading
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: <4.0,>=3.12
Requires-Dist: fastapi<0.116.0,>=0.115.13
Requires-Dist: httpx<0.29.0,>=0.28.1
Requires-Dist: numpy<3.0.0,>=2.0.0
Requires-Dist: omegaconf<3.0.0,>=2.3.0
Requires-Dist: pydantic<3.0.0,>=2.11.7
Requires-Dist: requests<3.0.0,>=2.32.4
Requires-Dist: tinydb<5.0.0,>=4.8.2
Requires-Dist: uvicorn<0.35.0,>=0.34.3
Provides-Extra: notebook
Requires-Dist: ipytest<1.0,>=0.14; extra == 'notebook'
Description-Content-Type: text/markdown

# phigrade

The PhiGrade client library. Instructors use it to publish an answer key by running
their reference solution; students use it to check their own code against that answer
key and get immediate, per-checkpoint feedback.

The defining property is that **student code never leaves the machine it runs on.**
Tests execute locally; what travels to the server is the *value* a function produced
at a checkpoint, which the server compares against the value the instructor's solution
produced at the same checkpoint. See the [top-level README](../README.md) for the full
rationale and the platform's data model.

This package works against a hosted PhiGrade backend or entirely offline against a
local file, using the same test code either way.

> **A word on vocabulary.** This document says **test** for the pytest function you
> write, and **unit check** for the record the server stores about it — the
> `UnitCheck` row that carries its point value and aggregation policy. The web
> application displays that same record to students as a **"Unit Test"**, and a
> `SubUnitCheck` as a **"Comparison"**. The code avoids the name `UnitTest`
> because pytest collects classes matching `Test*`, and these models are imported
> into test modules.

## Installation

```bash
pip install phigrade
```

Running from a Google Colab notebook instead? Install `phigrade[notebook]`, which pulls
in [ipytest](https://github.com/chmp/ipytest) — see [Colab notebooks](#colab-notebooks)
below.

For development inside this repository, the Python components form a single `uv`
workspace rooted at the repo:

```bash
uv sync --all-packages        # from the repository root
```

Requires Python 3.12+.

## Quick Start

**1. Write tests.** A PhiGrade test is an ordinary function decorated with `@weight`,
which declares its point value. Inside it, call a comparison helper at each point you
want checked.

```python
import numpy as np
import phigrade
from phigrade import weight

from mysolution import my_add, my_normalize


@weight(1.0)
def test_my_add():
    phigrade.is_equal(my_add(2, 3))


@weight(3.0)
def test_my_normalize():
    result = my_normalize(np.array([3.0, 4.0]))
    phigrade.all_close(result, rtol=1e-5, atol=1e-8)
```

Note what is absent: no expected values. The instructor's reference run supplies them.
The same file is the assignment's test suite *and* its answer key generator.

**2. Add a `phigrade.yaml`** next to the test file:

```yaml
course_id: "your-course-uuid"
assessment_id: "your-assessment-uuid"
use_local_server: false
server_url: "https://phigrade.example.org"
teacher_mode: false
timeout_seconds: 5
submission_files:
  - mysolution.py
```

Your API key is **never** written to `phigrade.yaml`. It is read from the
`PHIGRADE_API_KEY` environment variable, so a config file can be committed and shared
without leaking a credential:

```bash
export PHIGRADE_API_KEY="your-api-key"   # mint this on the Account Settings page
```

An `api_key` key in `phigrade.yaml` is a hard error pointing you at the environment
variable. Local mode needs no key at all.

**3. Run the tests.** They are named `test_*`, so `pytest` collects them normally:

```bash
pytest test_mysolution.py
```

Passing checkpoints print a confirmation; a failing checkpoint raises `AssertionError`
naming the checkpoint and showing what your code produced.

**4. Publish the answer key** (instructors). Run the same tests with
`PHIGRADE_TEACHER_MODE=true`, from a directory holding your reference solution. Each
checkpoint's value is recorded as the expected value, and the whole run publishes as
one definitive answer key when it finishes clean — see
[Teacher mode](#teacher-mode-teacher_mode-true) for what that means. The committed
`phigrade.yaml` stays `teacher_mode: false`, so there is nothing to set back:

```bash
PHIGRADE_TEACHER_MODE=true pytest test_mysolution.py
```

## Authoring API

Everything students and instructors need is exported at the package top level.

```python
import phigrade
from phigrade import weight, load_phigrade_config
```

The comparison helpers (`all_close`, `is_equal`, `row_match`, `key_value`,
`always_pass`) are called
qualified — `phigrade.all_close(...)` — so the module they come from is visible at the
call site. `weight` and `load_phigrade_config` are not comparison functions and are
imported bare.

### `@weight(w: float, aggregation: str = "fail_fast")`

Marks a function as a graded test and declares its point value. Attaches
`is_utest`, `autograder_function`, `autograder_module`, `weight`, and `call_count` to
the wrapped function, and installs the output-capture and Gradescope-finalization
wrappers.

`aggregation` decides how the test's checkpoints combine into its score, and is
validated at decoration time against `fail_fast` (the default), `even_weight`, and
`weighted`. See [Multiple checkpoints per test](#multiple-checkpoints-per-test).

> **The decorated function must be a module-level global.** The comparison helpers
> locate their calling test by walking the stack and looking each frame's function
> name up in that frame's module globals. Nested functions, methods, functions built
> by a factory whose `__name__` diverges from the name they are bound to, and calls
> made outside any test are all not found, and raise `PhiGradeUsageError`:
> `row_match() must be called from inside a module-level function decorated with
> @weight` — the message names all four cases.

### `phigrade.is_equal(system_output: Any, weight: float = 1.0) -> None`

Records `system_output` as a checkpoint, compared by exact equality (`==`). The value
must be JSON-serializable.

### `phigrade.all_close(system_output: np.ndarray, rtol=1e-05, atol=1e-08, equal_nan=False, weight=1.0) -> None`

Records a NumPy array checkpoint, compared with `np.allclose` at the given tolerances,
which are stored alongside the reference so the student run uses the same ones. Raises
`TypeError` — `all_close() expects a numpy ndarray, got list` — if `system_output` is
not an `ndarray`.

### `phigrade.always_pass(weight: float = 1.0) -> None`

A checkpoint that simply records `True`, equivalent to `phigrade.is_equal(True)`. Useful
when reaching a line of code is itself the thing being graded — for example, after an
in-test assertion that would have raised.

### `phigrade.row_match(system_output: str, weight: float = 1.0) -> None`

A **partial-credit** checkpoint for multi-line string output. Both sides are split into
lines and compared position by position, ignoring trailing whitespace; the fraction
awarded is `matched / max(len(reference_lines), len(produced_lines))`, so missing and
extra lines are both penalized. Raises `TypeError` — `row_match() expects a str, got
list` — if `system_output` is not a `str`.

> A partial score still *fails* the unit test: like every other helper, this one raises
> `AssertionError` unless the checkpoint earns full credit. The partial points are
> recorded on the server and appear in the student's score, and the `AssertionError`
> reports the score as a percentage of the checkpoint so partial credit is not mistaken
> for zero.

### `phigrade.key_value(system_output: str, threshold=1e-3, pattern=r"^(\S+)\s+(\S+)$", weight=1.0) -> None`

A **partial-credit** checkpoint for a file of `key value` lines — a metrics file, say.
Each non-blank line is parsed with `pattern` into a key and a numeric value, and a key
matches when it is present on both sides and the two values differ by less than
`threshold`, an **absolute** tolerance. The fraction awarded is
`matched / max(reference_keys, produced_keys)`, so a missing key, an extra key and a
malformed line each cost one key rather than the whole checkpoint. `threshold` and
`pattern` are recorded with the reference, so the student run reuses the instructor's
settings. Raises `TypeError` — `key_value() expects a str, got dict` — if
`system_output` is not a `str`.

Unlike `row_match`, keys are matched **by name, not by position**, and the
`AssertionError` names the keys that cost the points along with the values your code
produced for them:

```
1 of 2 keys matched. Values differed for: error(test) (yours: 0.55).
```

> It never names the reference value or the size of the difference. Either would hand
> back the answer key, which is the one thing that must stay on the server.

The `weight` argument on every helper is that checkpoint's relative weight, used only
by the `weighted` aggregation. It is recorded when the instructor publishes the answer
key, so students do not need to pass it.

All helpers set `__tracebackhide__`, so pytest hides the phigrade frames and the
failure points at your test.

### What a failed checkpoint looks like

```
AssertionError: Comparison 'test_education2_metrics@0' failed (scored 50% of this checkpoint).
1 of 2 keys matched. Values differed for: error(test) (yours: 999.0).
--- your output ---
error(train): 0.375000
error(test): 999.000000
--- end of your output ---
```

Three parts, in order:

1. **The headline** names the checkpoint and the score. The score is a percentage
   **of that checkpoint**, not of the test: checkpoints are always published with a
   maximum of `1.0` and the server scales the combined fraction to the test's
   `@weight`, so a checkpoint's share of the test's own points depends on the
   aggregation and on the other checkpoints — neither of which the client knows.
   Rounding never reports `100%` or `0%` for a score that was neither.
2. **The server's feedback**, when the comparison has any. `key_value` names the keys
   that cost the points; `row_match` and the all-or-nothing comparisons say nothing,
   and the message reads correctly without this line.
3. **Your output**, in a delimited block. It is whatever the checkpoint was given —
   a string is shown raw, a number, list or dict is rendered readably, and anything
   long is truncated in the middle. The block is closed and the message has no
   trailing newline, so a runner that appends the student program's own stdout and
   stderr underneath composes cleanly.

Nothing in any of it comes from the answer key. That is the rule the whole design
turns on: a student knows their own value, so even a *difference* would hand back the
reference.

### Multiple checkpoints per test

A test may call the helpers as many times as it likes. Each call becomes a separate
checkpoint named `<function>@<n>` — `test_my_concat@0`, `test_my_concat@1`, and so on —
so a single test can award partial credit.

How the checkpoints combine into the test's score is chosen with `@weight(...,
aggregation=...)` and recorded on the unit check when the answer key is published:

Each checkpoint scores a fraction between 0 and 1; the combined fraction is then
scaled to the test's declared point value, so a `@weight(3.0)` test with every
checkpoint correct earns 3.0.

| Aggregation | Test score |
| --- | --- |
| `fail_fast` (default) | `w ×` the smallest checkpoint fraction; correct only if every checkpoint is correct |
| `even_weight` | `w ×` the mean checkpoint fraction |
| `weighted` | `w ×` the mean weighted by each checkpoint's `weight` argument |

> The counter lives on the function object and increments across calls within a
> process. Invoking the same test twice in one interpreter produces `@0, @1` on the
> first run and `@2, @3` on the second, which will not match the reference. Let pytest
> run each test once.

### `load_phigrade_config(test_func) -> PhiGradeConfig`

Returns the resolved configuration, letting a test branch on mode — useful when the
reference run and the student run should exercise different code:

```python
config = load_phigrade_config(test_my_add)

@weight(3.0)
def test_handles_edge_case():
    if config.teacher_mode:
        phigrade.is_equal(reference_impl(...))
    else:
        phigrade.is_equal(student_impl(...))
```

## The Two Modes

The mode comes from `teacher_mode` in `phigrade.yaml`, or from the
`PHIGRADE_TEACHER_MODE` environment variable, which overrides it. There is no CLI flag.

The override is what lets the committed config stay `teacher_mode: false` forever:
publishing is then a property of the command, not an edit-and-revert dance on a file
that is easy to commit in the wrong state. It announces itself on stdout — see
[Environment overrides](#environment-overrides).

### Teacher mode (`teacher_mode: true`)

1. Creates a **staged reference attempt**. `submission_files` are sent as
   *paths with empty contents*, so the reference solution's source is never
   uploaded. The attempt is not the answer key yet — the previous key stays in
   force until the run completes.
2. Registers each `@weight` test as a unit check — an **identity** row keyed
   on the test's module and function name. Re-running is safe; an
   already-existing unit check is tolerated, not updated.
3. Declares *this run's* point value and aggregation for the test, staged
   with the reference attempt, and records each checkpoint's value and
   comparison settings as the expected answer.
4. **Publishes at the end of a clean run**, and only then. A publish is
   definitive: the answer key in force becomes exactly the tests and
   checkpoints that run recorded. A test you deleted stops being graded; a
   test whose module you renamed is a new test, and the old one retires.
   Earlier references are kept for the record and never graded against.
5. **Never asserts.** A reference run cannot fail a comparison.
6. Writes no Gradescope output and does not capture stdout.

> **A publish is definitive about *which* tests are in the key, but never
> retroactive about what already-graded work was scored out of.** Step 2
> tolerates an already-existing unit check rather than updating it — a unit
> check's *identity* is first-publish-wins, so renaming a test (or its module)
> is still the only way to give it a fresh identity — but its point value is
> not stored on that identity row at all: `@weight` is recorded fresh with
> *this* reference attempt every time it publishes, so editing `@weight(2)` to
> `@weight(5)` and republishing does take effect, the moment the new run
> activates. What does not happen automatically is retroactive: an attempt
> already graded keeps the denominator it was actually scored out of, even
> after a later publish changes what new submissions are scored against. To
> bring existing submissions onto the key now in force — because a `@weight`
> changed, or a reference output was wrong — call
> `phigrade.regrade_assessment()` after publishing. It re-runs the autograder
> over every stored submission's output, so it repairs a bad reference output
> the same way it repairs a bad `@weight`, and leaves any manual grading
> untouched.
>
> ```python
> if phigrade.finalize_reference_run():
>     phigrade.regrade_assessment()
> ```
>
> `regrade_assessment()` authenticates the same way every other call in this
> module does — with `PHIGRADE_API_KEY` — and the server route accepts it, so
> this runs from the same teacher-mode run that just published. It is staff
> only: a key whose owner is a student is refused. Against local mode the call
> succeeds too — the local server has no auth at all — but it typically has
> nothing to regrade, for an unrelated reason; see the note on local-mode
> regrade a few paragraphs down.

> **A run that dies partway publishes nothing.** A collection error, a raise
> in the reference solution or a `^C` leaves the staged attempt unpublished
> and the previous answer key in force, and the run says so at exit. Fix the
> problem and run it again — there is no half-published state to undo.
>
> Publishing happens through a pytest plugin that ships with the package (the
> `pytest11` entry point in `pyproject.toml`), so a plain
> `PHIGRADE_TEACHER_MODE=true pytest` needs nothing extra — as long as the
> installed package knows about the entry point. An environment set up before
> this shipped needs `uv sync --all-packages` (or a plain reinstall) once, so
> pytest actually discovers the plugin; run it before the plugin is installed
> and the exit warning above still catches you — every test recorded cleanly,
> but you get "this teacher-mode run did not complete" anyway, because nothing
> ever called the publish. If you drive the tests yourself rather than through
> pytest — a custom runner, a notebook — call `phigrade.finalize_reference_run()`
> yourself once they have all run; it is the same idempotent publish the
> plugin calls, safe to call again on a run that already published.

> **Teacher mode refuses to run under `pytest-xdist` at all.** Each `-n`
> worker is a separate process running only part of the suite, so no single
> process ever holds the whole answer key; publishing from one would drop
> every test the others ran. `PHIGRADE_TEACHER_MODE=true pytest -n auto` fails
> immediately, on the first test, before anything is staged. `-n auto` is
> still fine — and, for a large class, wanted — for grading *student* runs
> and for the `pytest -n auto` case mentioned under
> [Submission Slots](#submission-slots); it is only a teacher-mode publish
> that must run single-process. Drop `-n` (or add `-p no:xdist`) to publish.

In local mode, a teacher run **rebuilds the assessment it is publishing** from
scratch: every row belonging to that `assessment_id` is dropped when the
reference attempt arrives, so a checkpoint or a test you deleted cannot
survive in the file. Rows belonging to *other* assessments are untouched,
which is what lets one answer-key file hold several
[submission slots](#submission-slots). This is deliberately unlike the
server, which supersedes rather than deletes: offline there is no gradebook,
and nothing to keep history for. Offline still follows the same grading rule
— a key that was never published does not grade — so a local run that dies
partway leaves a file that reports a missing reference rather than one that
grades against half a key. An answer-key file written by an older `phigrade`
has no record of which reference attempt wrote each reference and will not
grade at all; the 404 it produces says so and points at republishing, since
there is deliberately no legacy fallback.

**Local-mode regrade is inert in the normal split-process workflow.** Student
mode always runs against a throwaway temp copy of the shared answer-key file
(`local_server_db_file`, see `_setup_db`) and never writes back to it, so
that file never accumulates the student attempts a regrade would act on —
`regrade_assessment()` against local mode will typically report zero attempts
regraded even right after a student run that should need one. The local
server's `/regrade` route exists for contract parity with the real backend
and is exercised by tests that drive one teacher-mode server directly; it is
not reachable end to end through the ordinary "teacher publishes, student
runs, instructor regrades" sequence offline.

### Student mode (`teacher_mode: false`)

1. Creates a normal attempt, uploading the **actual contents** of every path in
   `submission_files`. A missing file raises `FileNotFoundError`, naming the directory
   that was searched — `submission_files` resolve against the current working
   directory, so running pytest from the wrong place is the usual cause.
2. Submits each checkpoint's produced value and reads back the verdict.
3. Prints a line per checkpoint; raises `AssertionError` on the first failure within a
   test.
4. Writes Gradescope output if configured.

Students cannot define unit checks. A checkpoint with no published reference raises,
which is the expected signal that the instructor has not published the answer key yet.

## Colab notebooks

Coding labs can run entirely inside a Google Colab notebook — tests, solution, and
submission all in cells, with no local Python install. phigrade does not stop using
pytest to do this: [ipytest](https://github.com/chmp/ipytest) runs pytest inside the
notebook kernel, driving phigrade exactly as a plain `pytest` invocation would.

**1. Install and configure ipytest**, once per notebook session:

```python
!pip install 'phigrade[notebook]'

import ipytest

import phigrade
from phigrade import weight

ipytest.autoconfig()
```

**2. Write `phigrade.yaml`.** A Colab session starts with an empty disk, so write it
from a cell instead of uploading it:

```python
%%writefile phigrade.yaml
course_id: "your-course-uuid"
assessment_id: "your-assessment-uuid"
use_local_server: false
server_url: "https://phigrade.example.org"
submit_colab_notebook: true
```

`phigrade.yaml` and `submission_files` both resolve against the working directory in a
notebook — `/content` — so the two can never land in different directories the way they
can on a laptop, and the "Required file not found" message's usual cause (pytest
started from the wrong place) cannot arise here.

**3. Write and run tests.** `%%ipytest` must be the first line of the cell:

```python
%%ipytest
@weight(1.0)
def test_my_add():
    phigrade.is_equal(my_add(2, 3))
```

Calling `test_my_add()` directly, without `%%ipytest`, does not work — phigrade looks
for `phigrade.yaml` next to the test's file, and a cell has no file until ipytest gives
it one for the duration of the run. Calling the test directly raises
`PhiGradeUsageError` saying so and pointing back at this section, rather than the
confusing `TypeError` a bare notebook cell would otherwise produce.

**If the assessment uses [submission slots](#submission-slots) or `group_members`, use
`%%ipytest -s` instead.** Submitting to a slot prints `Submitting to: <slot name>`, and
a group submission prints `Submitting as a group: <emails>`, both from inside the test —
and both are academic-integrity matters: which slot a submission landed in, and who it
was submitted for, should be visible, not swallowed. Plain `%%ipytest` captures that
output, so a passing test shows only `1 passed` with no confirmation of which slot was
used or who the group was; `-s` disables the capture.

### Authentication

Add a Colab secret named `PHIGRADE_API_KEY`: open the key icon in the left sidebar,
add a secret with that name, paste in a key minted from the web app's Account Settings
page, and turn on **Notebook access** for this notebook.

Use the secret rather than pasting the key into a cell. A Colab secret is
per-Google-account and granted per-notebook, so it is **not** shared when the notebook
is shared. A key typed into a cell travels with the notebook to everyone it is shared
with — and, with `submit_colab_notebook` on, would be uploaded to the server inside the
submission along with everything else in the notebook.

phigrade resolves the API key in this order: the `PHIGRADE_API_KEY` environment
variable, then the Colab secret of the same name, then the credentials file from
`phigrade auth login`. The environment variable stays first so nothing outside a
notebook changes.

### `submit_colab_notebook`

Set `submit_colab_notebook: true` in `phigrade.yaml` and a student's run attaches the
running notebook to the submission, keyed by its file name and stored as a plain
`.ipynb` file.

**Outputs are stripped before upload.** Cell outputs are the only part of a notebook
that gets large — a single embedded matplotlib plot is 50-200 KB of base64 — and the
default per-assessment attempt-file budget is 512000 bytes for everything a submission
carries. A notebook with a handful of plots could blow that budget and be rejected at
the worst possible moment, so only code and markdown are uploaded.

**Teacher mode never attaches the notebook.** Teacher mode already uploads
`submission_files` as names with empty contents, because the reference solution must
never reach the database — and the instructor's notebook *is* the reference solution,
so it is never captured at all.

**It is an error outside Colab.** Running with `submit_colab_notebook: true` anywhere
but a Colab notebook raises `PhiGradeUsageError`, naming the config key and saying to
set it `false` for local runs, rather than silently submitting without the notebook the
student believes they sent.

> **Restart the session before the run you want recorded.** phigrade caches
> the notebook it attaches, the submission it creates, and `phigrade.yaml`
> itself for the life of the kernel — a plain `pytest` process is fresh every
> time, but a notebook kernel is not. The first `%%ipytest` cell you run in a
> session captures the notebook and the submission at that moment; editing
> code and re-running `%%ipytest` afterwards submits again, but the attached
> `.ipynb` and the config are still the ones from that first run, not your
> latest edits. Before the run you actually want graded, use **Runtime →
> Restart session and run all** so the notebook, the submission, and
> `phigrade.yaml` are all captured fresh.

> **Publish the answer key from the notebook, not from a `.py` file.** A
> test defined in a cell belongs to module `__main__`, and the server
> identifies a unit check by its module *and* function name. A key
> published from `test_lab1.py` records the module `test_lab1`, so a
> student's notebook run looks up `__main__`, finds nothing, and is told
> the answer key is missing. Publish from the same notebook instead — a
> shell prefix like `!PHIGRADE_TEACHER_MODE=true ...` does nothing here,
> since a notebook cell has no shell to prefix and that would only set the
> variable for a subprocess. Set it on the kernel itself, in a cell
> **above** the `%%ipytest` cell:
>
> ```python
> import os
> os.environ["PHIGRADE_TEACHER_MODE"] = "true"
> ```
>
> or set `teacher_mode: true` in the `%%writefile phigrade.yaml` cell.
> Either way, use **Runtime → Restart session and run all** (see the
> stale-notebook warning above) so the setting is in effect before the
> tests run.

## Configuration

The only configuration source is a YAML file named `phigrade.yaml`, located **in the
directory containing the test module** that defines the `@weight` function — not the
current working directory. It is loaded once and cached for the life of the process.

Five keys can be overridden from the environment — see
[Environment overrides](#environment-overrides) below. No other key can.

| Key | Type | Default | Notes |
| --- | --- | --- | --- |
| `course_id` | str | **required** | Required in both local and remote mode |
| `assessment_id` | str or mapping | **required** | The assessment's UUID, or a submission-slot block — see [Submission Slots](#submission-slots) |
| `use_local_server` | bool | `false` | Run against a local file instead of a server; `PHIGRADE_USE_LOCAL_SERVER` overrides |
| `server_url` | str | — | Required when remote; overwritten in local mode |
| `teacher_mode` | bool | `false` | Reference run vs. student run; `PHIGRADE_TEACHER_MODE` overrides |
| `timeout_seconds` | int | `2` | Per-request HTTP timeout |
| `local_server_db_file` | str | `phigrade_db.json` beside the config | The local answer key; `PHIGRADE_LOCAL_DB_FILE` overrides |
| `gradescope_json_file` | str | — | Enables Gradescope output (student mode only) |
| `submission_files` | list[str] | — | Paths **relative to the working directory** |
| `group_members` | list[str] | — | Every student the submission is for, **including you**; requires the assessment's group size to allow it |
| `submit_colab_notebook` | bool | `false` | Attach the running notebook to the submission — see [Colab notebooks](#colab-notebooks); error outside Colab |

The API key is deliberately absent from this table: it is read from the
`PHIGRADE_API_KEY` environment variable, and an `api_key` key in the file is rejected.

Remote mode requires `server_url`, `course_id`, `assessment_id`, and a non-empty
`PHIGRADE_API_KEY`; local mode requires `course_id` and `assessment_id`. Note that
`submission_files` resolve
against the CWD while `phigrade.yaml` resolves against the test module's directory —
the usual cause of a `FileNotFoundError` is running pytest from the wrong directory.

### Environment overrides

| Variable | Overrides | Values |
| --- | --- | --- |
| `PHIGRADE_TEACHER_MODE` | `teacher_mode` | `1`/`true`/`yes`, `0`/`false`/`no` |
| `PHIGRADE_USE_LOCAL_SERVER` | `use_local_server` | the same |
| `PHIGRADE_LOCAL_DB_FILE` | `local_server_db_file` | a path |
| `PHIGRADE_SLOT` | which slot `assessment_id` resolves to | a `slot_name` |
| `PHIGRADE_GROUP_MEMBERS` | `group_members` | comma-separated email addresses |

Between them, one committed `phigrade.yaml` serves online publishing, offline
development, and a separate offline answer key per slot.

The two booleans are matched case-insensitively and ignore surrounding whitespace. Any
*other* value is an error naming the variable, rather than a silent default: quietly
reading a misspelt `ture` as false would fail to publish an answer key with no
indication of why. An unset or blank variable is simply not an override.

`PHIGRADE_LOCAL_DB_FILE` is used exactly as given, so a relative path resolves against
the **working directory** — unlike the config key it overrides, which defaults to a path
beside `phigrade.yaml`.

Every override announces itself on stdout when the config loads:

```
Teacher mode: enabled (PHIGRADE_TEACHER_MODE)
Local answer key: slot_a_key.json (PHIGRADE_LOCAL_DB_FILE)
```

That line is the whole point of announcing: a variable exported once in a shell profile
must not silently republish an answer key, or redirect a submission, weeks later. The
config is loaded once per process, so each line appears at most once per `pytest` run.

### Submission Slots

`assessment_id` can be a mapping instead of a single UUID string, to offer several
assessments as named **submission slots** the student picks between — for example a
"human work only" slot beside a "human or AI work" slot, both backed by their own
assessment on the server:

```yaml
course_id: "your-course-uuid"
assessment_id:
  message: "Select the appropriate submission slot."
  slots:
    - slot_name: "Human only"
      slot_description: "This submission contains human work only."
      slot_assessment_id: "human-only-assessment-uuid"
    - slot_name: "Human or AI"
      slot_description: "This submission may contain AI-assisted work."
      slot_assessment_id: "human-or-ai-assessment-uuid"
```

`message` is optional and defaults to `"Select the appropriate submission slot."`.
Each entry of `slots` requires all three keys: `slot_name`, `slot_description`, and
`slot_assessment_id` (the assessment UUID that slot resolves to).

The slot is resolved once, at config-load time, in this order:

1. The `PHIGRADE_SLOT` environment variable, if set to a non-empty value. The match
   against `slot_name` ignores surrounding whitespace and case.
2. Otherwise, an interactive prompt listing each slot's name and description,
   numbered.
3. Otherwise, a `ValueError` naming the valid slot names.

Either way the resolved slot is announced — `Submitting to: Human only` — so a
`PHIGRADE_SLOT` exported once in a shell profile cannot silently redirect later
submissions. On the environment-variable path that line goes to stdout, which
pytest captures and prints on failure (or always, under `-s`).

The prompt reads and writes `/dev/tty` directly rather than stdin/stdout, so it
works under a plain `pytest` run **with no `-s` flag** — pytest's output capture
never sees it.

Some runs have no terminal that can be prompted on, and they **must** set
`PHIGRADE_SLOT` to one of the slot names:

```bash
export PHIGRADE_SLOT="Human only"
```

That covers non-interactive environments — a Gradescope autograder, CI — and two
cases that would otherwise hang: `pytest -n auto` (each xdist worker is its own
process, so N workers would interleave N prompts on one terminal) and any
background job (`nohup pytest &` under a job-control shell, where reading the
terminal raises `SIGTTIN` and suspends the process). Both are detected and turned
into the same error naming the valid slot names.

Because the config is loaded once and cached for the life of the process (see
above), a `pytest` run resolves the slot — and therefore prompts, if it prompts at
all — at most once, no matter how many tests run.

#### Teacher mode publishes to one slot only

A reference run (`teacher_mode: true`) resolves the slot exactly as a student run
does, and then publishes the answer key to that **one** slot's assessment —
references are per-assessment on the server. A student who picks any other slot
gets a `Sub-unit-check reference not found` error on every checkpoint, naming the
assessment, module, function and checkpoint it looked for.

So an instructor must publish once per slot. Because the config is cached per
process, that means one process per slot:

```bash
PHIGRADE_TEACHER_MODE=true PHIGRADE_SLOT="Human only" pytest test_mysolution.py
PHIGRADE_TEACHER_MODE=true PHIGRADE_SLOT="Human or AI" pytest test_mysolution.py
```

Run those before releasing the assignment, and re-run both whenever the reference
solution changes.

In local mode both publishes can share one answer-key file: a reference run clears only
the rows of the assessment it is publishing. Giving each slot its own file with
`PHIGRADE_LOCAL_DB_FILE` also works, and is what you want when the two keys are
distributed separately.

## Errors

Everything phigrade raises deliberately is a `PhiGradeError`, so a runner can tell
"phigrade could not do its job" from "the student's code is wrong" — the latter
arrives as an `AssertionError` and nothing else.

| Exception | Raised when |
| --- | --- |
| `PhiGradeUsageError` | A comparison helper is called where no `@weight` test can be found |
| `PhiGradeConnectionError` | The server could not be reached: DNS, refused connection, or `timeout_seconds` |
| `PhiGradeAuthError` | 401 or 403 — the API key is missing, wrong, expired, or not enrolled |
| `PhiGradeNotFoundError` | 404 — the assessment, unit check, checkpoint or answer key is not there |
| `PhiGradeServerError` | 5xx — the server failed |
| `PhiGradeResponseError` | Any other unexpected status; the base of the three above, and it carries `status_code` |

They are classified by **status code**, never by the server's message text. The two
"not found" 404s — a unit check and a reference — differ only in their prose, and that
prose is rewritten whenever it can be made clearer; code matching on it would break
every time a message improved. What each 404 actually was is in the message, which is
written to be read rather than parsed.

**A submission past an assessment's late due date is a `409`, not a `403`.** It falls
through to `PhiGradeResponseError`, carrying the server's own message naming the
cutoff — deliberately not `PhiGradeAuthError`, whose "the API key is missing, wrong,
expired, or not enrolled" would send a student who submitted an hour late looking at
the wrong problem. An assessment the caller cannot see at all — before its release
date — is still a `403`, since that really is a permission the caller lacks.

Older releases raised a bare `Exception` for all of these.

## Authenticating

Remote mode needs an API key. There are two ways to provide one:

**`PHIGRADE_API_KEY`** — set it in the shell before running tests, as shown above.
Good for CI and for one-off overrides.

**`phigrade auth login`** — a saved credential, so you don't need the environment
variable in every shell:

```bash
phigrade auth login --server https://phigrade.example.org
```

It prompts for an API key (mint one in the web app's Account Settings page),
validates it against the server, and on success saves it to
`$XDG_CONFIG_HOME/phigrade/credentials.json` (`~/.config/phigrade/credentials.json`
if `XDG_CONFIG_HOME` is unset), created with file mode `0600` so it is never
world-readable. `phigrade auth whoami --server <url>` reports which account a saved
(or `--api-key`-supplied) credential belongs to. `--token` is still accepted as an
alias for `--api-key`, for scripts written before the terminology settled.

A remote-mode `phigrade.yaml` run resolves its key in this order: `PHIGRADE_API_KEY`
first, then the credentials file for that `phigrade.yaml`'s `server_url`. The
environment variable always wins, so it can still override a saved credential without
disturbing it.

## Local Mode

Set `use_local_server: true` and no backend is needed. The client starts a local server
in a daemon subprocess on an automatically chosen port, backed by a TinyDB JSON file.

This is not a mock: the local server implements the same HTTP API as the hosted
backend, and both use the *same* comparison code (`phigrade/compare.py`, which the
backend imports directly). A suite developed locally behaves identically against a real
course. Local mode has no authentication, so `PHIGRADE_API_KEY` need not be set.

Two distinct uses:

* **Drafting an assignment** before a course exists on the server.
* **Distributing a self-contained assignment** — ship the tests plus the answer-key
  JSON, and students get immediate feedback with no account, network, or credentials.

**The invariant that makes distribution safe:** a student-mode run never writes to the
answer-key file. It requires the file to exist, copies it to a temporary file, and
serves from the copy. Running the tests cannot corrupt or reveal the key, accidentally
or otherwise. (`tests/test_student_mode_no_persistence.py` exists to enforce this.)

## Gradescope Integration

Set `gradescope_json_file` in student mode and the client writes a Gradescope
`results.json`:

```yaml
gradescope_json_file: "results.json"
```

Behavior worth knowing:

* The file is rewritten after **every** test, so it is valid even if the run is killed
  partway through.
* `stdout` and `stderr` inside a test body are tee-captured — still shown live — and
  included in that test's output. Python streams only; subprocess output is not caught.
* A test that raises before making any comparison is still reported, with an
  explanatory message and a score of zero.
* Unit checks the student never attempted appear with score zero, since the test list
  comes from the published unit-check set rather than from what ran.
* A test is `passed` only if it made *all* its expected comparisons and all passed —
  skipping comparisons cannot yield a pass.
* Individual outputs longer than 500 characters are truncated in the middle.

No `run_autograder` / `setup.sh` scaffolding ships with this package; wire it into your
own Gradescope container.

## Package Layout

```
phigrade/
├── phigrade/
│   ├── __init__.py            Public exports: weight, is_equal,
│   │                          all_close, always_pass,
│   │                          row_match, key_value, PhiGradeConfig,
│   │                          load_phigrade_config, the PhiGradeError
│   │                          hierarchy, __version__
│   ├── phigrade.py            The client: config loading, the @weight decorator and
│   │                          its wrappers, comparison helpers, submission creation,
│   │                          HTTP calls, Gradescope finalization, finalize_reference_run
│   ├── pytest_plugin.py       The pytest11 plugin: publishes a staged reference
│   │                          attempt from pytest_sessionfinish on a clean run
│   ├── app.py                 create_app() — the TinyDB-backed local server, mirroring
│   │                          the backend's REST API
│   ├── server.py              Server lifecycle: port discovery, subprocess spawn,
│   │                          readiness polling
│   ├── compare.py             Scoring and aggregation — shared with the backend
│   ├── messages.py            The not-found messages — shared with the backend, so
│   │                          local mode and the hosted server cannot diverge
│   ├── errors.py              The exception hierarchy raised by the client
│   ├── gradescope_output.py   results.json writer
│   ├── cli.py                 The `phigrade` console script: `auth login`, `auth whoami`
│   ├── credentials.py         The CLI's credentials file (server-keyed API keys)
│   └── colab.py               Everything that only works inside Google Colab: fetching
│                              and stripping the running notebook, the secrets-based API
│                              key lookup
├── examples/                  A runnable end-to-end example (see below)
└── tests/                     The suite (see below)
```

### Examples

`examples/` is both documentation and a regression test — its tests run as part of the
default suite.

| File | What it shows |
| --- | --- |
| `simplefns.py` | The code under test, including a deliberately wrong variant |
| `test_simplefns.py` | The canonical autograder file: single and multiple comparisons per test, branching on `teacher_mode`, and the missing-reference error |
| `phigrade.yaml` | A local, student-mode configuration |
| `phigrade_db.json` | A checked-in answer key from a teacher run — this is what makes the example runnable out of the box |
| `run_teacher_mode.py` | The reference workflow, including the global-state reset ritual |

Run them from the `phigrade/` directory (not from `examples/`, since paths resolve
against the CWD):

```bash
uv run --all-packages pytest examples/test_simplefns.py
```

## Contributing

### Tests

```bash
cd phigrade
uv run --all-packages pytest
uv run --all-packages pytest tests/test_local_server.py
```

There is no `conftest.py` and no pytest configuration — discovery is the default from
the `phigrade/` directory.

| Test file | Covers |
| --- | --- |
| `test_local_server.py` | The local server's HTTP endpoints directly |
| `test_is_equal_local.py` | The largest suite: end-to-end runs against generated test modules, call-count tracking, Gradescope output, error paths |
| `test_all_close_local.py` | Tolerances, non-array inputs, mixed comparison types |
| `test_compare_rowmatch.py` | `rowmatch` partial-credit fractions and aggregation validation, at the unit level |
| `test_compare_keyvalue.py` | `keyvalue` partial-credit fractions and its per-key feedback, at the unit level |
| `test_phigrade_aggregation_local.py` | Each aggregation type end-to-end, plus `always_pass`, `rowmatch` and `keyvalue` |
| `test_phigrade_config.py` | Config loading and the environment overrides (the invalid-config paths are **not** covered) |
| `test_student_mode_no_persistence.py` | The answer-key isolation invariant, and the per-assessment reset |
| `test_teacher_mode_endpoints.py` | Which endpoints each mode calls, with `requests.post` mocked |
| `test_notebook_config_errors.py` | The error a notebook user gets when tests are not run under `%%ipytest` |
| `test_notebook_ipytest.py` | Every phigrade feature exercised from notebook cells through ipytest — the Colab support guarantee, in executable form |
| `test_colab.py` | `colab.py` exercised without Colab, by injecting a fake `google.colab` module |
| `test_colab_api_key.py` | The API key resolution order in a notebook: environment, then Colab secret, then credentials file |
| `test_submit_colab_notebook.py` | `submit_colab_notebook`: config parsing, output stripping, the outside-Colab error, and that teacher mode never attaches the notebook |
| `test_real_backend_integration.py` | Full stack against a real backend instance |

The last of these boots the actual backend, which works because the `uv` workspace puts
both packages in one environment. `notebook_harness.py` is not itself a test file — it
runs notebook cells in a real IPython shell, in a subprocess, and is what
`test_notebook_ipytest.py` uses to drive them.

### Global state

The client keeps process-global mutable state: `_config`, `_local_server`,
`_submission_id`, `_unit_check_attempt_ids`, `_unit_check_definition_cache`,
`_gradescope_test_errors`, and `_gradescope_test_output`. There is no public reset API,
so anything that runs more than one logical session in a single interpreter must clear
these by hand — see the `clear_phigrade_state` autouse fixture in
`test_real_backend_integration.py` and the reset in `examples/run_teacher_mode.py`.
Removing this global state in favor of an explicit session object would be a welcome
improvement.

### Lint, format, and types

```bash
make python-tools-phigrade      # from the repo root: ruff check, ruff format, mypy
```

Ruff handles both linting and formatting (it replaced black). Line length 88, target
`py312`, rules `E,F,I,B,UP`. Type hints on everything; docstrings on public functions.

### Releasing

`make upload` from the repository root — it refuses to run with a dirty working tree,
then builds and uploads to PyPI:

```bash
make upload      # uv build --package phigrade && twine upload dist/*
```

Bump `version` in `pyproject.toml` first; `phigrade.__version__` reads it from package
metadata at runtime.

## Known Limitations

* **The CLI is auth-only.** `phigrade auth login` and `phigrade auth whoami` are the
  only `phigrade` console-script commands; there is no `phigrade run` or similar, and
  tests are still run with `pytest`.
* **Only five keys have environment-variable overrides** — see
  [Environment overrides](#environment-overrides). Everything else is YAML only.
* **Checkpoint maximums are always `1.0`** when references are published. That is by
  design — a checkpoint scores a fraction, and the fraction is scaled to the test's
  `@weight` — but it does mean a checkpoint cannot carry its own point value. Relative
  weighting between checkpoints is settable, via each helper's `weight` argument under
  `weighted` aggregation.
* **`Server` has no `stop()`.** The subprocess is a daemon and dies with its parent.
* **Student-mode temporary answer-key copies are never cleaned up.**
* **The comparison set is small** — exact equality, `np.allclose`, `row_match` and
  `key_value`. Adding a comparison type means changing `compare.py`, which the backend
  imports, so both components must be updated and released together.

## License

MIT. See [LICENSE](./LICENSE).
