Metadata-Version: 2.5
Name: judgements
Version: 0.2.1
Summary: TypeSafe System One for pydantic users: declare questions as fields of a model, get a model back.
Project-URL: Homepage, https://github.com/nk412/judgements
Project-URL: Repository, https://github.com/nk412/judgements
Project-URL: Issues, https://github.com/nk412/judgements/issues
Author: Nagarjuna Kumarappan
License-Expression: MIT
License-File: LICENSE
Keywords: classification,judgements,probabilities,pydantic,system-one,typesafe
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Requires-Dist: pydantic>=2
Requires-Dist: typesafe-sdk>=0.7
Requires-Dist: typing-extensions>=4
Description-Content-Type: text/markdown

# judgements

TypeSafe's System One model answers narrow, typed questions about a piece of state and returns
calibrated probabilities instead of text. `judgements` wraps that API so it works with Pydantic models:
you declare the questions as fields of a model, and you get a model back.

See `hello.py` for it in action.

## Quickstart

```
pip install judgements
```

## Run hello.py

You will need to set `TYPESAFE_API_KEY` in your environment.

```
python hello.py
```

```
Triage(billing=True p=0.98, tone=frustrated p=1.00, urgency=today p=0.98)
route to billing
```

## The pydantic model is the set of questions

```python
class Triage(Judgements):
    billing: bool = question("Is this ticket about billing?")
    tone: Tone = question("What is the customer's tone in `body`?")
    urgency: Urgency = score("How urgent is this ticket?")

triage = Triage.ask(ticket)
```

Each field poses a question, and the annotation sets the answer type. `Triage.ask(state)` sends every field in a single request and returns a `Triage` whose fields hold the answers.

There are three kinds of question, depending on the annotation.

| Annotation | Kind | What comes back |
| --- | --- | --- |
| `bool` | noul, "does this hold?" | `True` or `False` |
| an `Enum` or `Literal["a", "b"]` | choice, "which one?" | the chosen member |
| an `Enum` with `score(...)` | score, "how much, on this scale?" | the most probable level |

`question(...)` infers noul or choice from the annotation.

For an Enum, member names are the labels and member values are their descriptions. Write descriptions as the concrete situations you mean:

```python
class Urgency(Enum):
    can_wait = "No deadline is implied; handle in the normal queue"
    this_week = "The customer expects a resolution within a few days"
    today = "The customer is blocked or demands immediate action"
```

For a score, order matters: the first member is level 0. A `Literal` gives labels without descriptions.

## Reading the answer

Fields are plain values, so `triage.tone == Tone.angry` and `if triage.billing:` just work. The probabilities are one attribute away:

```python
triage.p.billing            # 0.98, probability that the answer is yes
triage.p.tone               # {Tone.calm: 0.0, Tone.frustrated: 1.0, Tone.angry: 0.0}
triage.confidence.tone      # 1.0, confidence in the chosen option
triage.expected.urgency     # 1.97, probability-weighted level from 0 to 2
triage.results["urgency"]   # ScoreResult(can_wait 0.01, this_week 0.01, today 0.98; expected 1.97)
triage.usage                # Usage(requests=1, input_tokens=312, output_tokens=48)
```

`expected` exists for score fields. It is the probability-weighted average of the level positions, and the number to use when ranking or averaging many items. The field itself holds the most probable level.

Use the probabilities to make policy explicit:

```python
if triage.billing and triage.p.billing > 0.9:
    route_to_billing()
elif triage.confidence.tone < 0.6:
    escalate_to_human()
```

`triage.model_dump()` gives `{'billing': True, 'tone': 'frustrated', 'urgency': 'today'}`.

The names `p`, `confidence`, `expected`, `results`, `usage`, `questions`, `ask` and `from_answers` are reserved and cannot be fields.

## Clients

`Triage.ask(ticket)` uses a default client that reads `TYPESAFE_API_KEY`. For anything beyond a script, make one:

```python
ts = TypeSafe()                                   # or AsyncTypeSafe(), then `await ts.ask(...)`

triage = ts.ask(ticket, Triage)
triage, refund = ts.ask(ticket, Triage, wants_refund)      # several things, one request
triages = ts.map(tickets, Triage)                          # one request per ticket, in order
ts.usage                                                   # requests and tokens so far
```

State can be a pydantic model, a dict, a list or a string. It is sent as JSON as is, so backticked paths in instructions, like `` `body` ``, refer into the state.

`ask` also takes `model=`, `retry=` and `timeout=`. The async `map` takes `concurrency=`, eight by default.

## Questions on their own

A question does not need a model. On its own it returns the full result:

```python
wants_refund = question("Does the customer explicitly ask for money back?", bool)
r = ts.ask(ticket, wants_refund)      # NoulResult(no, p=0.08)
bool(r), r.probability
```

Options can be decided per request, which is how you rerank or select among candidates. A list gives labels, a dict adds descriptions:

```python
best = choice("Which of `candidates` best answers `query`?", candidates)
r = ts.ask({"query": query, "candidates": candidates}, best)
r.choice, r.ranked                    # the winner, and every candidate by probability

relevance = score("How relevant is `text` to `query`?", {"none": "Off topic", "partial": "Related", "direct": "Answers it"})
r = ts.ask({"query": query, "text": text}, relevance)
r.level, r.score, r.at_least("partial")
```

## Writing good questions

- Ask one narrow judgement per field. Fields are answered in parallel in the same request.
- Put the judgement in the instructions and the possible answers in the type. The field name is not shown to the model.
- Include a way out when nothing may fit, such as an `unclear` or `other` member.
- Check the exact request with `request(ticket, Triage)` before spending tokens.

## Testing without a key

```python
from judgements.testing import FakeTypeSafe

fake = FakeTypeSafe(billing=0.9, tone=Tone.angry, urgency={Urgency.today: 0.7, Urgency.this_week: 0.3})
triage = fake.ask(ticket, Triage)     # same parsing path as the real client, no network
fake.requests[0]["state"]             # what would have been sent
```

Answers are matched by field name: a probability or bool for a `bool` field, an option or a dict of option to probability for the others. `AsyncFakeTypeSafe` does the same for async code.

## Further reading

The TypeSafe docs cover the model itself: [System One](https://docs.typesafe.ai/concepts/system-one), [state](https://docs.typesafe.ai/concepts/state), [the three primitives](https://docs.typesafe.ai/primitives) and [confidence](https://docs.typesafe.ai/confidence).
