Metadata-Version: 2.4
Name: polarity-keystone
Version: 0.3.3
Summary: Python SDK for the Polarity agent evaluation + sandboxed-execution platform (legacy package name: keystone)
Author: Polarity
License: MIT
Project-URL: Homepage, https://plr.sh
Project-URL: Repository, https://github.com/Polarityinc/keystone-sdk-python
Project-URL: Issues, https://github.com/Polarityinc/keystone-sdk-python/issues
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 4 - Beta
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Dynamic: author
Dynamic: requires-python

# Polarity SDK for Python

Python client for the [Polarity](https://plr.sh) agent evaluation +
sandboxed-execution platform. Shares a single source of truth with the
TypeScript and Go SDKs — byte-identical cost estimates and prompt
rendering across all three.

> **Rebrand note**: this SDK is now called Polarity. The previous import
> name (`polarity_keystone`) and class name (`Keystone`) continue to work
> indefinitely, so existing code does not need to change. New code should
> prefer `from polarity import Polarity`.

## Install

```bash
pip install polarity-keystone
```

The pip package name is unchanged; only the public import name changed.
Zero external runtime dependencies — uses only `urllib` from the
standard library.

## 60-second quick start: `Eval()`

The shortest path from "I have an agent" to "I have an evaluation":

```python
from polarity import Eval, Factuality, AnswerRelevancy

result = Eval(
    "summarisation-quality",
    data=[
        {"input": "Long article about whales...", "expected": "Whales are mammals."},
        {"input": "Article about Java GC...",     "expected": "Java GC reclaims memory."},
    ],
    task=lambda input: my_agent(input),                # your agent / prompt
    scores=[
        Factuality(model="paragon-fast"),
        AnswerRelevancy(),
    ],
    max_concurrency=4,
)

print(result.summary)                                   # p50/p95/mean per scorer
```

If `POLARITY_API_KEY` (or the legacy `KEYSTONE_API_KEY`) is set, the run
is also recorded to your dashboard; otherwise it stays purely local.

## Build a dataset from production traces

Turn failing prod traces into a regression eval set in one call:

```python
from polarity import Polarity

plr = Polarity()
result = plr.datasets.from_traces(
    name="summarize-regressions",
    filter={"tool": "summarize", "since": "24h"},
)
print(result["rows_added"], result["sample_rows"])
```

Idempotent on `span_id` within a single call. Events missing `input` /
`expected` are skipped and counted under `result["skipped"]`. Pass
`dry_run=True` to inspect the rows without writing.

## Pytest plugin: eval-as-tests

Once the package is installed, pytest auto-loads a `polarity_score`
fixture (the legacy `keystone_score` name keeps working):

```python
from polarity.scorers import Factuality

def test_summary(polarity_score):
    out = my_agent("Long text…")
    polarity_score(Factuality(expected="Short summary"),
                   input="Long text…", output=out, min=0.8)
```

If `min` is set and the score falls below it, the test fails. At
session end the plugin posts one eval per test module to the dashboard
(`POLARITY_API_KEY` required; pass `--polarity-no-flush` — or the legacy
`--keystone-no-flush` — to skip the post).

## Sandbox-as-a-tool ergonomics

`create()` / `get()` / `list()` return a bound **`SandboxHandle`** so an
agent loop can call the sandbox without threading the ID:

```python
sb = plr.sandboxes.create(spec_id="spec-123")

sb.exec("python script.py")
sb.write("/tmp/input.json", json.dumps(payload))
out = sb.read("/tmp/output.json")
diff = sb.diff()

sb.destroy()
```

Same pattern on **`ExperimentHandle`** and **`AgentSnapshotHandle`**:

```python
exp = plr.experiments.create("nightly", spec_id="s")
results = exp.run_and_wait(scores=[Factuality(), ExactMatch(expected_key="expected")])
cmp = exp.compare(other_exp)                            # handle or string ID
m   = exp.metrics()

snap = plr.agents.upload("codex", "./bundle", entrypoint=["python3", "agent.py"])
snap.delete()
```

## Auto-instrument every LLM client at once

```python
from polarity import auto_instrument

auto_instrument(sandbox_id=os.environ.get("POLARITY_SANDBOX_ID"))
# Patches every installed provider in-place. Subsequent OpenAI/Anthropic/
# Mistral/Google/LiteLLM/Claude Agent SDK/DSPy/LangChain calls auto-trace.
```

Wraps **OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent
SDK, DSPy, LangChain** in one call — every prompt, token count, and tool
call shows up in your dashboard with no other code changes.

## Manual tracing when you want it

```python
from polarity import traced

# 1. Decorator
@traced
def fetch_user(uid: str) -> dict:
    return db.users.find(uid)

# 2. Named decorator
@traced(name="planning-phase")
def plan(prompt: str) -> str:
    return llm.complete(prompt)

# 3. Context manager
with traced(name="embed-doc"):
    openai.embeddings.create(...)
```

Spans automatically nest using contextvar-based parent linking — no need
to plumb a context object through your code.

## Environment variables

Both the new Polarity-branded names and the legacy Keystone-branded
names are honored; the new name wins when both are set.

| Preferred | Legacy (still works) | Purpose |
| --- | --- | --- |
| `POLARITY_API_KEY` | `KEYSTONE_API_KEY` | API key for authentication |
| `POLARITY_BASE_URL` | `KEYSTONE_BASE_URL` | Override the default server URL |
| `POLARITY_SANDBOX_ID` | `KEYSTONE_SANDBOX_ID` | Injected by the platform inside sandboxes |

Default server URL: `https://plr.sh`.

API keys begin with `plr_live_…` going forward; existing `ks_live_…`
keys continue to be accepted by the server.

## What's in the SDK

- **9 client services** — `sandboxes`, `specs`, `experiments`, `alerts`, `agents`, `datasets`, `scoring`, `export`, `prompts`
- **3 bound handles** — `SandboxHandle`, `ExperimentHandle`, `AgentSnapshotHandle` with delegated methods
- **29 built-in scorers** across 5 families:
  - **Heuristic** (6): `ExactMatch`, `Levenshtein`, `NumericDiff`, `JSONDiff`, `JSONValidity`, `SemanticListContains`
  - **LLM-judge** (9): `Factuality`, `Battle`, `ClosedQA`, `Humor`, `Moderation`, `Summarization`, `SQLJudge`, `Translation`, `Security`
  - **RAG** (8): `ContextPrecision`, `ContextRecall`, `ContextRelevancy`, `ContextEntityRecall`, `Faithfulness`, `AnswerRelevancy`, `AnswerSimilarity`, `AnswerCorrectness`
  - **Embedding** (1): `EmbeddingSimilarity`
  - **Sandbox invariants** (5): `FileExists`, `FileContains`, `CommandExits`, `SQLEquals`, `LLMJudge`
- **`@Scorer` decorator** — wrap any `(scenario) -> score` function
- **`Eval(name, data, task, scores)`** — Braintrust-style one-call eval primitive
- **`@traced` decorator** + context manager — auto-capture spans with contextvar-based propagation
- **Client wrapping** — `plr.wrap(openai_client, sandbox_id=...)` adds transparent trace reporting (sync / async / streaming)
- **`auto_instrument()`** — single call patches every installed provider (OpenAI, Anthropic, Mistral, Google GenAI, LiteLLM, Claude Agent SDK, DSPy, LangChain)
- **Prompt management** — `plr.prompts.create(slug, template)` / `plr.prompts.get(slug)` / `Prompt.render(**vars)` with server-side audit + byte-identical local renderer
- **Bulk export** — `plr.export.traces(...)`, `.spans(...)`, `.scenarios(...)`, `.scores(...)` auto-paginate; `plr.export.experiment(id)` for full JSON or NDJSON dumps
- **OpenTelemetry bridge** — `wrap()` emits `gen_ai.*` metadata; `register_otel_flush()` piggy-backs on program shutdown

## Versioning

Semver. `0.x` = alpha/beta while the API shape settles.

## License

MIT.
