Metadata-Version: 2.4
Name: goldset
Version: 0.1.0
Summary: Turn production traffic into a versioned, deduplicated, coverage-tracked regression suite — a curation accelerator, not a gold factory.
Project-URL: Homepage, https://github.com/AshwinUgale/goldset
Project-URL: Repository, https://github.com/AshwinUgale/goldset
Project-URL: Issues, https://github.com/AshwinUgale/goldset/issues
Author: Ashwin Ugale
License: MIT License
        
        Copyright (c) 2026 Ashwin Ugale
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: clustering,coverage,curation,drift,evaluation,golden-set,llm,observability,pii,regression-testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: numpy>=1.21
Requires-Dist: pyyaml>=6.0
Provides-Extra: cluster
Requires-Dist: hdbscan>=0.8; extra == 'cluster'
Requires-Dist: scikit-learn>=1.0; extra == 'cluster'
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=2.2; extra == 'embeddings'
Provides-Extra: pii
Requires-Dist: presidio-analyzer>=2.2; extra == 'pii'
Requires-Dist: presidio-anonymizer>=2.2; extra == 'pii'
Description-Content-Type: text/markdown

# goldset

[![CI](https://github.com/AshwinUgale/goldset/actions/workflows/ci.yml/badge.svg)](https://github.com/AshwinUgale/goldset/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.10%E2%80%933.12-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

> **A curation accelerator for regression test sets** — it turns your production traffic into a
> versioned, deduplicated, coverage-tracked golden set, with a mandatory human confirmation step so
> the model's own past mistakes never get enshrined as ground truth. It is a curation *accelerator*,
> **not** a gold factory.

**[View the live demo report](https://ashwinugale.github.io/goldset/)** — the planted-defect
validation suite (does the tool recover known problems?), generated by `goldset demo` and published
on every push.

`goldset` reads a log of real inputs and past model outputs, clusters them into intents, mines the
failures and rare cases frequency-sampling would miss, deduplicates the rest, and walks you through
confirming an expected outcome for each — then exports a golden suite your eval runner can execute.
Coverage, drift, an enforced dev/regression/holdout split, and PII redaction come built in.

The one idea everything is built around: **you cannot bootstrap gold from ungraded logs.** A logged
output may be wrong; auto-accepting it bakes the model's errors into your benchmark forever. So
`goldset` *proposes* candidates and shows the logged output as a *suggestion* — a human (or a
genuinely trusted oracle) confirms the expected outcome, and nothing is exported as golden until
they do.

## Limitations (read first)

1. **It accelerates curation; it does not manufacture ground truth.** Mined logs give you inputs and
   the model's *past* outputs, which may be wrong. A human sets each expected outcome — the tool
   only makes that fast.
2. **Coverage is relative to your observed traffic, not absolute.** A taxonomy derived from traffic
   can only "cover" intents that already appeared; behaviours users never tried are invisible. Pass
   an external taxonomy (`--taxonomy`) to measure coverage against a spec instead.
3. **Edge-case mining quality is capped by the outcome signals in your logs.** With no thumbs-down /
   escalation / error / cost signals, only frequency-based mining is possible — and the tool says so
   rather than pretending it found the hard cases.
4. **A holdout only stays honest if it stays uninspected.** `goldset` locks role changes, counts
   holdout inspections, and warns when the holdout is compromised — but it cannot enforce discipline
   once data leaves it.

## Quick start

The demo runs a keyless, self-checking validation suite end to end — no API key, no model download:

```bash
pip install git+https://github.com/AshwinUgale/goldset.git
goldset demo --html demo.html
```

`demo` plants known problems (duplicates, a rare escalation, a coverage gap, an unconfirmed case, a
locked holdout) and proves the tool recovers each — the known-answer test the whole design rests on.
It exits non-zero if any check misbehaves, so it doubles as a smoke test in CI.

## The workflow

Each command reads and writes one JSON workspace, so the pipeline is resumable:

```bash
goldset ingest ./logs.jsonl --out ws.json     # 1. read logs; redact PII BEFORE anything is stored
goldset cluster --workspace ws.json           # 2. embed + cluster inputs into candidate intents
goldset select  --workspace ws.json           # 3. medoid + boundary + MMR + mined failures, deduped
goldset label   --workspace ws.json           # 4. REQUIRED: confirm the expected outcome per case
goldset roles   --workspace ws.json --lock    # 5. deterministic dev/regression/holdout split, locked
goldset export  --workspace ws.json --out suite.yaml   # 6. golden promptfoo suite
```

At step 6, export is **blocked** until step 4 has confirmed outcomes — a suite with unconfirmed
cases is a *candidate* set, not golden, and the exporter refuses to label it otherwise.

Ongoing, once a suite exists:

```bash
goldset coverage --workspace ws.json                      # relative coverage of the derived intents
goldset coverage --workspace ws.json --taxonomy spec.yaml # absolute coverage vs an external spec
goldset drift    ./new_traffic.jsonl --workspace ws.json  # new intents not yet represented
goldset mine     --workspace ws.json                      # signal-bearing entries not yet confirmed
goldset version  --workspace ws.json --bump minor --reason "added 12 cases"
```

## The golden gate

A case becomes **golden** only when both are true:

- it is **confirmed** — a human or trusted oracle set the expected outcome (`trust_tier` is
  `human_confirmed` or `oracle_confirmed`, not `unconfirmed`); and
- it carries a **contract** — a set of assertions, or (for deterministic tasks) an exact expected
  output.

Until then it is a *candidate*. `goldset export` emits only golden cases and fails closed (exit `2`)
if nothing is confirmed; `--include-candidates` will emit the unconfirmed ones but marks the whole
artifact `golden: false` so it can never be mistaken for a trusted set.

## Cases are assertions by default

For generative systems an exact reference output is usually the wrong contract, so the default case
is a set of assertions over the output and behaviour:

```yaml
input: "how do I cancel my subscription?"
assertions:
  - must_mention: cancellation_date
  - must_not_claim: refund_completed
  - tool_called: { name: lookup_subscription }
  - final_state: { status: cancellation_pending }
```

Exact-output matching (`equals`) is reserved for deterministic tasks.

## Input format

A log is a JSON array or JSONL — one record per line:

```json
{"input": "how do I cancel?", "output": "Go to settings to cancel.",
 "signals": {"thumbs_down": true, "escalated": false}, "timestamp": 1723000000}
```

Only `input` is required. Common field spellings are accepted as aliases (`prompt` / `query` /
`question` → `input`; `response` / `completion` / `answer` → `output`; `ts` / `created_at` →
`timestamp`). `signals` drives failure mining and priority preservation — negative feedback,
escalations, tool errors, safety flags, and cost are all recognized. PII in `input` / `output` is
redacted before the record is ever written to the workspace.

## Export targets

The first exporter targets [promptfoo](https://promptfoo.dev). Text assertions map to native
promptfoo checks (`must_mention` → `icontains`, `must_not_*` → `not-icontains`, `equals` →
`equals`, `regex` → `regex`); structural assertions (`tool_called`, `final_state`) have no native
promptfoo check without a provider hook, so they are carried in each test's `metadata` and
disclosed — never silently dropped. Each exported test also carries its `goldset_id`, `trust_tier`,
and `set_role` for traceability.

### Stress-testing a suite with muteval

Because the export is standard promptfoo, a goldset suite feeds straight into
[muteval](https://pypi.org/project/muteval/) to ask *"how good is this suite?"* — muteval mutates the
system under test and reports which cases catch the regressions. Add your prompt (the mutation
target) to goldset's exported `tests` and point muteval's promptfoo adapter at it:

```yaml
# promptfooconfig.yaml
prompts: ["You are a support agent. ... {{input}}"]   # your system under test
providers: ["openai:gpt-4o-mini"]                      # your model
tests: !include suite.yaml#tests                       # goldset's exported tests
```

```bash
muteval --promptfoo promptfooconfig.yaml
```

All of goldset's assertion types (`icontains`, `not-icontains`, `equals`, `regex`) are in muteval's
supported set, so the hand-off is lossless — the only thing you add is the prompt and provider, which
are your system, not the dataset.

## Validated on real data

Beyond the keyless `demo` (a planted-defect known-answer suite), goldset is validated on the public,
PII-safe [Bitext customer-support corpus](https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset)
(27 intents / 11 categories). Its gold intents serve as both a clustering ground truth and an
external taxonomy:

| embedder | clusters (26 intents) | category purity | coverage vs gold taxonomy | held-out category flagged as drift |
|---|---|---|---|---|
| keyless hashing (core) | 94 | 0.83 | 93% | 48% |
| sentence-transformers (`[embeddings]`) | 47 | **0.99** | 96% | 62% |

Clusters respect the coarse intent structure (0.99 category purity with real embeddings), coverage
against the 27-intent spec is 93–96%, and a held-out category is correctly surfaced as novel traffic.
Reproduce with [`examples/validate_on_bitext.py`](examples/README.md#real-run-validation-validate_on_bitextpy).

## Library

```python
from goldset import ingest_file, cluster_entries, select_candidates, export_promptfoo_yaml

ws = ingest_file("logs.jsonl")               # redacted LogEntry corpus in a Workspace
ws.clustering = cluster_entries(ws.entries)  # intent clusters
ws.suite.extend(select_candidates(ws).cases) # unconfirmed candidate cases
# ... confirm outcomes (goldset.labeling) ...
yaml_text = export_promptfoo_yaml(ws.suite)  # raises NothingGoldenError until confirmed
```

See [`examples/`](examples/) for a complete, runnable end-to-end script on a sample support-bot log.

## Development

```bash
pip install -e ".[dev]"
python -m pytest
ruff check .
```

The core is dependency-light (`numpy` + `pyyaml` + stdlib) and the whole test suite is deterministic
and offline — a keyless hashing embedder and a greedy clusterer stand in for the heavy ML stack.
Real sentence-transformers embeddings, HDBSCAN clustering, and Presidio NER redaction live behind
the opt-in `[embeddings]`, `[cluster]`, and `[pii]` extras. Python 3.10–3.12.

## License

MIT — see [LICENSE](LICENSE).
