Metadata-Version: 2.4
Name: evaluableai
Version: 0.4.0
Summary: EvaluableAI — the native software implementation of the R-BED™ methodology for governable AI systems.
Author: Vishal Kumar Srivastava, Sah
License: Proprietary
Project-URL: Homepage, https://evaluable.ai
Keywords: AI governance,R-BED,model risk,evaluation,materiality,assurance
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.23
Requires-Dist: scikit-learn>=1.1
Requires-Dist: pandas>=1.5
Provides-Extra: piml
Requires-Dist: PiML>=0.6; extra == "piml"
Provides-Extra: modeva
Requires-Dist: modeva; extra == "modeva"
Provides-Extra: docs
Requires-Dist: nbformat; extra == "docs"
Requires-Dist: jupyter; extra == "docs"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Provides-Extra: all
Requires-Dist: PiML>=0.6; extra == "all"
Requires-Dist: nbformat; extra == "all"
Requires-Dist: jupyter; extra == "all"
Requires-Dist: pytest>=7; extra == "all"

# EvaluableAI

**EvaluableAI is the native software implementation of the R-BED™ methodology for building governable AI systems.**

R-BED™ is the *methodology and ontology*. EvaluableAI is the *engine* that operationalizes it: it evaluates an AI system across the full R-BED chain, generates evidence under the canonical thirteen sub-dimensions (each with explicit provenance), composes a materiality tier, reports evidence sufficiency and gaps separately from risk, and produces a governance-ready decision.

> R-BED™ is introduced in *Evaluable AI: Building Governable AI Systems* (Srivastava & Sah, 2026). R-BED™ is a trademark of Vishal Kumar Srivastava. EvaluableAI computes its technical evidence natively with `numpy` and `scikit-learn`; PiML and MoDeVa are **optional** cross-check / import sources only — never runtime dependencies, and they never define R-BED.

---

## The R-BED chain

```
R-BED
  ├── Vertex 1 — Technical Failure           P(F)
  │     System Design · Data Foundations · Failure Modes · Fairness ·
  │     Stability & Reliability · Safety · Explainability
  ├── Vertex 2 — Deployment Harm             P(H | F, A)
  │     Human Mediation · Exposure to Harm · Harm Detection & Control
  └── Vertex 3 — Business Impact             E[S | H, F, A]
        Propagation · Severity · Recovery

  sub-dimension ─▶ evidence requirements ─▶ diagnostics / structured facts / checked artifacts
                ─▶ sub-dimension assessment ─▶ materiality ─▶ governance decision
```

The thirteen sub-dimensions are the **canonical, frozen** top level of the ontology. Diagnostics (calibration, robustness, weak-region analysis, disparity, …) are a *lower-level evidence layer* underneath them — a single sub-dimension aggregates **many** evidence items of mixed provenance, and no diagnostic is ever treated as a silent proxy for a whole sub-dimension.

## What EvaluableAI keeps separate (by design)

| Concept | Meaning |
|---|---|
| **Raw metric** | The number a diagnostic computed (e.g. AIR = 0.59). |
| **Interpretation** | What that number means in context. |
| **Concern** (0–3 ordinal) | A severity level. **Not a probability.** |
| **Evidence sufficiency** | How well-supported the sub-dimension is — reported **separately from risk**. |
| **Vertex score / materiality tier** | The composed result, with a full, inspectable trace. |
| **Provenance** | Computed · Structured · Evidence-checked · Imported · Pending. |

Vertex 2 and Vertex 3 are **never fabricated** from model metrics — they require structured deployment / business input, and are marked `Pending` (with an explicit gap) until supplied.

## Install

```bash
pip install evaluableai
# optional cross-check / import adapters (never required):
pip install "evaluableai[piml]"      # import existing PiML results as evidence
pip install "evaluableai[modeva]"    # import existing MoDeVa results as evidence
```

Runtime dependencies: `numpy`, `scikit-learn`, `pandas`. Nothing else.

## Quick start

```python
import evaluableai as eai

a = eai.RBEDAssessment("Adult public-data demo", system_type="predictive_ml")

a.set_system_spec({
    "intended_task": "income screening",
    "intended_users": "credit analysts",
    "decision_role": "advisory",
    "operating_envelope": "US tabular applicants",
    "prohibited_uses": "no automated adverse action",
    "known_assumptions": "stationary population",
    "data_provenance": "OpenML Adult (id 1590)",
})

a.set_model(
    model,
    train={"X": X_train, "y": y_train},
    calibration={"X": X_cal, "y": y_cal},
    test={"X": X_test, "y": y_test},
    sensitive={"sex": sensitive_test},
)

# Vertex 2 / 3 are structured facts about the deployment — never inferred:
a.set_deployment({
    "human_mediation": "auto_with_override",
    "exposure_to_harm": "bounded_external",
    "controls": {"live_outcome_monitoring": True, "subgroup_monitoring": True,
                 "override_logging": True, "tested_halt": False},
})
a.set_business_impact({"propagation": "portfolio", "severity": "material",
                       "recovery": "costly"})

a.plan()      # what evidence each sub-dimension needs, and what is missing
a.run()       # gather evidence across all 13 sub-dimensions

a.report()                    # the 13 canonical R-BED rows (concern + provenance + sufficiency + gaps)
a.evidence_report()           # every individual evidence record
a.evidence_gaps()             # what is missing, per sub-dimension
a.evidence_sufficiency()      # sufficiency per sub-dimension — separate from risk

a.materiality().explain()     # full scoring trace (weights, cutoffs, vetoes, tier)
a.decision()                  # governance action combining tier + sufficiency + vetoes
print(a.governance_report())  # narrative, board-ready report
```

## Scoring is an explicit author decision

No canonical numeric weights or cutoffs were available to hard-code, so EvaluableAI ships **two named, versioned, non-canonical placeholder configs** and refuses to pretend either is official:

- `legacy_additive_v0` — additive 20/40/40 vertex weighting on a 0–3 composite (default).
- `expected_loss_multiplicative_v0` — a conceptual P(F) × P(H|F,A) × E[S] product of normalized vertex means.

**Every** `Materiality` result carries `author_decision_required=True` and a loud note. Hard vetoes can only *escalate* a tier, never lower it. See `docs/SCORING.md` and `docs/AUTHOR_DECISIONS.md`.

## Different architectures, same ontology, different evidence

The same thirteen sub-dimensions evaluate predictive ML, RAG, and agentic AI. The
*evidence plan* and *diagnostics* underneath adapt to the system type; the
ontology does not move.

```python
a = eai.RBEDAssessment("RAG demo", system_type="rag")
a.set_rag_trace(trace, corpus_stats=cs)      # retrieval/grounding/citation evidence
a.run(); a.report()                          # same 13 rows

a = eai.RBEDAssessment("Agent demo", system_type="agentic_ai")
a.set_agent_trajectories(trajectories)       # task/planning/tool/trajectory evidence
a.run(); a.report()                          # same 13 rows
```

`plan()` returns the architecture-appropriate evidence plan; `Failure Modes` asks
for weak-region analysis on a predictive model, retrieval quality on a RAG system,
and planning/tool/trajectory failures on an agent — all under the same
sub-dimension. RAG and agent evidence is generated by native diagnostics in
`evaluableai.diagnostics.rag` / `.agents`, and two fully offline reference systems
(`evaluableai.systems`) let the public-data demos run with no model download and
no API key (the agent executes GSM8K's own calculator annotations). Vertex 2/3
stay structured/PENDING for every architecture — public data never fabricates
deployment or business facts. See `docs/CHANGELOG_CROSS_ARCHITECTURE.md`.

## Backward compatibility

The previous `Assessment` API, `build_registry`, and the `PILLARS_BY_VERTEX` / `ALL_PILLARS` symbols still work, re-implemented on top of the R-BED engine. The historical name *"Stability"* now resolves to the canonical **"Stability & Reliability"**, and *"pillar"* is a deprecated alias for *"sub-dimension"*. See `docs/MIGRATION.md`.

## Layout

```
evaluableai/
  ontology.py            # canonical, frozen R-BED ontology (13 sub-dimensions, 3 vertices)
  rbed/                  # the evidence-to-decision engine
    technical_failure/   deployment_harm/   business_impact/
    evidence.py  assessment.py  scoring.py  decision.py  api.py  profiles.py
  diagnostics/           # lower-level evidence generators (native numpy/sklearn)
  integrations/          # optional sklearn/custom + PiML/MoDeVa import adapters
  reporting/             # dataframe / json / governance report
  data/                  # machine-readable ontology (JSON + YAML)
tests/                   # ontology, evidence, diagnostics, mapping, scoring, e2e
docs/                    # ARCHITECTURE, EVIDENCE_MODEL, SCORING, AUTHOR_DECISIONS, MIGRATION, CHANGELOG
notebooks/               # public-data end-to-end demo
```

## License

Proprietary. R-BED™ is a trademark of Vishal Kumar Srivastava.
