Metadata-Version: 2.4
Name: kestrel-feature-observability
Version: 0.17.12
Summary: Observability feature for Kestrel Sovereign — a per-agent emitter that emits OTel spans for hook-driven lifecycle events, with a shared Prometheus registry
Project-URL: Homepage, https://kestrelsovereign.com
Project-URL: Source, https://github.com/KestrelSovereignAI/kestrel-feature-observability
Project-URL: Issues, https://github.com/KestrelSovereignAI/kestrel-feature-observability/issues
Author: UncleSaurus
Maintainer: UncleSaurus
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,kestrel,metrics,observability,prometheus
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: <3.15,>=3.11
Requires-Dist: httpx>=0.27.0
Requires-Dist: kestrel-sovereign-sdk<1,>=0.34
Requires-Dist: openinference-semantic-conventions>=0.1.9
Requires-Dist: opentelemetry-exporter-otlp-proto-http<2,>=1.20
Requires-Dist: opentelemetry-sdk<2,>=1.20
Provides-Extra: fleet
Requires-Dist: kestrel-sovereign-sdk<1,>=0.34; extra == 'fleet'
Provides-Extra: metrics
Requires-Dist: kestrel-sovereign-sdk[metrics]<1,>=0.34; extra == 'metrics'
Provides-Extra: test
Requires-Dist: graphql-core>=3.2; extra == 'test'
Requires-Dist: kestrel-sovereign-sdk[metrics]<1,>=0.34; extra == 'test'
Requires-Dist: pytest-asyncio>=1.1.0; extra == 'test'
Requires-Dist: pytest>=8.0.0; extra == 'test'
Description-Content-Type: text/markdown

# kestrel-feature-observability

The **single** observability package for Kestrel Sovereign — one repo, one
version, one source of truth for the whole observability domain — with two roles
selected by the two entry-point groups (package boundaries need not match
process boundaries):

- **Base install** (`pip install kestrel-feature-observability`) — the
  lightweight per-agent **emitter** `Feature` (the `kestrel_sovereign.features`
  entry point). It attaches an `ObservabilityHook` to the agent's hook system;
  every lifecycle event is emitted as an OpenTelemetry span (a session
  `run_span` with child `tool_span`s) via `KestrelTracer`, exported over
  OTLP/HTTP to whatever `OTEL_EXPORTER_OTLP_ENDPOINT` points at (e.g. a
  host-supervised Phoenix). No DB — this is what every agent gets. Prometheus
  metrics emit through the SDK's shared registry, so a single `/metrics` scrape
  stays coherent across the framework + every feature package.
- **Host extra** (`kestrel-feature-observability[fleet]`) — enables the
  **`FleetObservabilityHostFeature`** (the `kestrel_sovereign.host_features`
  entry point), which ships the single "Observability" console panel with a
  three-view sub-nav: **Timeline** (default) answers when activity happened with
  a temporal overview and compact span detail; **Navigator** answers where it
  fits with the hierarchical Tenant → Fleet → Agent → Subagent → Session → Turn
  → Events tree and a persistent inspector for the selected Turn/Event span;
  and **Phoenix** provides exhaustive trace forensics in the curated thin embed.
  Timeline and Navigator links preserve the exact OTel span ID in both
  directions, visibly highlight it, and report an honest containing-Turn or
  no-highlight fallback when that exact span is unavailable. All three are pure
  read-models over Phoenix's GraphQL through the same-origin
  `/phoenix/graphql` proxy (no store or new host routes). The HostFeature lives
  in the `kestrel_feature_observability.fleet` subpackage. Since the custom
  store/entities were retired, `fleet/feature.py` imports only the
  `HostFeature`/`UIContributions` contract from `kestrel_sdk`, so the host role
  is gated by the **SDK version**, not by an extra-only importable module: the
  package declares the SDK floor-only (`>=0.34`) across every install role — the
  host pins the SDK minor, so this package must not add a second ceiling (#99). The import/entry point
  stays **guarded** — if the
  resolved SDK is too old to export the contract, it degrades to `None` (with a
  warning logged) and the host skips the panel instead of crashing the feature
  scan.

> **Embed note:** the browser console may log `No HydrateFallback element
> provided to render during initial hydration` on the Phoenix subtab — this
> comes from Phoenix's own React Router bundle (vendor-streamdown chunk) during
> SPA hydration and is expected upstream noise (cosmetic, no functional impact);
> the arize-phoenix bump that resolves it is blocked by kestrel-sovereign's
> `fastapi` pin.

> This package supersedes the separate `kestrel-feature-observability-fleet`
> package, which is deprecated.

## Installation

```bash
uv pip install kestrel-feature-observability
```

For real Prometheus output:

```bash
uv pip install 'kestrel-feature-observability[metrics]'
```

For the fleet host role (the Phoenix-embed console panel):

```bash
uv pip install 'kestrel-feature-observability[fleet]'
```

Both features are auto-discovered by Kestrel Sovereign via their entry-point
groups — install the base package alongside `kestrel-sovereign` and
`ObservabilityFeature` registers itself into every agent; install with `[fleet]`
on the host and `FleetObservabilityHostFeature` registers at host scope.

## Emitter transport

The hook emits OpenTelemetry spans via `KestrelTracer`
(`kestrel_feature_observability.tracing`), exported over OTLP/HTTP. Endpoint
discovery is OTel-standard:

- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` — a full traces endpoint (used as-is), or
- `OTEL_EXPORTER_OTLP_ENDPOINT` — a base endpoint (the exporter appends
  `/v1/traces`), e.g. the host-supervised local Phoenix.
- `OTEL_EXPORTER_OTLP_HEADERS` — honored for auth.
- `KESTREL_OTEL_PROJECT` — the Phoenix project spans land in, stamped as the
  `openinference.project.name` Resource attribute. Defaults to `kestrel-fleet`,
  which the fleet console's curated Observability panel deep-links to — so
  per-agent hook traces show up there instead of Phoenix's "default" project.

When no OTLP endpoint is configured the tracer is a **no-op** — no provider, no
exporter, no network — so the emit path costs nothing and the agent runs
unaffected (Prometheus counters still fire locally). The span shape is the nested
doll **session ⊃ turn ⊃ tool ⊃ tool-start markers**, one trace per turn (the
session band is an attribute grouping — OpenInference `session.id` and the
backward-compatible `kestrel.session_id` are stamped with the same value on
every span — not a trace). A session-marker root is exported lazily on the
first lifecycle event; each `UserPromptSubmit` starts a turn (a new trace root
`<agent> turn <n>`, tagged `kestrel.turn_id`/`kestrel.turn_index`); each
`PreToolUse` emits an instant `<tool> (started)` marker and each `PostToolUse` a
child `tool_span` (tool name, real duration, success,
`kestrel.tool_outcome=completed`) parented to the current turn; `Stop`
reconciles the turn's unfinished tools and emits a `turn <n> summary` (the
session stays live), and `AgentTerminate`/teardown emits the true `session
summary` aggregating turns. `orchestrator` is the agent itself when
self-driven, else inherited.

The `turn <n> summary` carries the per-turn stats — `kestrel.tool_count`,
`kestrel.error_count`, `kestrel.success_ratio`, `kestrel.denied_count`,
`kestrel.incomplete_count`, and `kestrel.duration_ms` (the go-forward unified
key; the legacy `kestrel.turn_duration_ms` is still emitted alongside for
back-compat) — and the `session summary` aggregates them across the session
(`kestrel.turn_count` plus the same totals, with `kestrel.duration_ms` alongside
the legacy `kestrel.session_duration_ms`). By default the turn root carries no
prompt text; setting `KESTREL_OTEL_CAPTURE_PROMPTS=1` opts in to stamping the
turn's user prompt on its root span as `input.value`, truncated to
`KESTREL_OTEL_MAX_IO_CHARS` (default `20000`) characters.

### Tool outcomes

Every tool call ends in exactly **one** terminal span, stamped with
`kestrel.tool_outcome`:

- `completed` — the tool ran: the `PostToolUse` (Claude Code hook: also
  `PostToolUseFailure`) `tool_span`, with its real duration and `tool.success`.
- `denied` — Claude Code's `PermissionDenied` event fired, i.e. the auto-mode
  permission classifier refused the call. Also carries
  `tool.denied_by="classifier"`. Claude Code hook only.
- `incomplete` — the tool never completed and no deny event fired: a
  `PreToolUse` guard, a permission rule, or an aborted turn. Produced by
  turn-end reconciliation.

A `denied`/`incomplete` span is a **zero-duration point span at the recorded
start**, carrying `tool.success=false`: the tool never ran, so its span never
claims runtime it did not have. It still pairs with its `<tool> (started)`
marker, so a refused tool renders as one visible terminal stub instead of an
orphaned open-ended band.

Turn-end reconciliation is the backbone and does not depend on a deny event
being available: on `Stop` — and, for a turn interrupted before its `Stop`, on
the next `UserPromptSubmit` or session close — every still-pending tool start
becomes a terminal `incomplete` span emitted into **its own** original turn,
never re-attributed to the current one. An interrupted turn also gets a
retroactive `turn <n> summary`, so every turn is summarized exactly once. The
Claude Code hook's 30-minute pending-start TTL sweep remains only as a backstop
for a session abandoned entirely.

"Exactly one terminal span" holds even when the events arrive out of order.
Claude Code runs each hook event as its own process, so the per-session lock
serializes state writes but guarantees nothing about lifecycle ordering: a
`PostToolUse` can land *after* `Stop` already reconciled its call. Reconciling —
and `PermissionDenied` — therefore leave a **tombstone** for the call (keyed by
`tool_use_id`, else the tool name; bounded by the same TTL/cap as the pending
starts, and by tool name in-process). A terminal event that finds no pending
start but does find a tombstone claims it and emits **nothing**: an exported
OTel span cannot be retroactively re-labeled `incomplete` → `completed`, so the
span already exported stands and the late duplicate is dropped rather than
double-counted and mis-parented into whatever turn is live by then. For the same
reason a completion is always parented, attributed and counted against the turn
stamped on its own `PreToolUse` start, never the current turn.

`kestrel.tool_count`/`kestrel.error_count`/`kestrel.success_ratio` stay over
**executed** tools only — a refusal is not an agent error — so the refusals ride
on the two additive keys `kestrel.denied_count` and `kestrel.incomplete_count`,
present on both the `turn <n> summary` and the `session summary`, and existing
dashboards keep their exact semantics. The fleet Timeline paints a
`denied`/`incomplete` span as a fixed-width labeled stub (`<tool> · denied`)
rather than the unreadable 2px tick its true extent implies — a refusal is an
event operators must see — while the span data stays honestly zero-duration.

The in-process emitter produces only `completed` and `incomplete`: the SDK's
`HookInput` has no per-call id (so pending starts pair by tool **name**, LIFO)
and no deny event, which makes a refusal indistinguishable from an abort. The
Claude Code hook has both — `tool_use_id` for exact pairing of concurrent
same-name calls, and `PermissionDenied` for the explicit `denied` label.

## Claude Code hook emitter

The same package ships a **`kestrel-obs-claude-hook`** console script so that
**Claude Code** sessions land in the fleet Observability Timeline exactly like
kestrel agents and talon runs. Claude Code's hooks system runs a shell command
per lifecycle event with a JSON payload on stdin; this script turns those events
into the identical span shape as the in-process emitter above — session ⊃ turn ⊃
tool ⊃ tool-start markers, one trace per turn — posted over OTLP/HTTP:

- `SessionStart` → an immediately-ended `AGENT` session-marker root
  (`session.id` and `kestrel.session_id` = the Claude session id,
  `kestrel.agent_name` = `claude-code`, `kestrel.orchestrator` =
  `$KESTREL_OBSERVABILITY_ORCHESTRATOR` else `Direct`). Both session attributes
  are retained on every span the hook emits.
- `UserPromptSubmit` → a labeled `claude-code turn <n>` root (a new trace). By
  default the root carries no prompt text; setting `KESTREL_OTEL_CAPTURE_PROMPTS=1`
  opts in to stamping the payload's `prompt` on the turn root as `input.value`,
  truncated to `KESTREL_OTEL_MAX_IO_CHARS` (default `20000`) characters.
- `PreToolUse` → an instant `<tool> (started)` marker under the current turn,
  and the start is recorded as pending — keyed by `tool_use_id`, which is also
  stamped as `tool.call_id` on the marker and on its terminal span so the
  Timeline pairs concurrent same-name calls one-to-one. `PreToolUse` fires
  *before* the permission check, so a tool attempt is visible the moment it
  starts, even one about to be refused.
- `PostToolUse` / `PostToolUseFailure` → a completed `tool_span`
  (`kestrel.tool_outcome=completed`) under the current turn. `PostToolUse` fires
  after a tool **succeeds**; failed tools fire the separate `PostToolUseFailure`
  event (top-level `error` / `duration_ms`), which is recorded as a failed span.
  Duration prefers the payload's own `duration_ms`, else the gap to the paired
  `PreToolUse`.
- `PermissionDenied` → a terminal `tool_span` with `kestrel.tool_outcome=denied`
  and `tool.denied_by="classifier"`, paired to its marker by `tool_use_id` and
  anchored zero-duration at the recorded start (see **Tool outcomes** above).
  Reconciliation would catch the call at `Stop` anyway; this event only upgrades
  the label from `incomplete` to `denied` and emits it sooner.
- `Stop` → turn-end reconciliation, then a `turn <n> summary` (the session stays
  live); a turn interrupted before its `Stop` is reconciled and summarized
  retroactively at the next `UserPromptSubmit` / `SessionEnd`. `SessionEnd` (and
  a defensive staleness sweep) → the true `session summary` and state cleanup.
  The turn summary carries `kestrel.tool_count`, `kestrel.error_count`,
  `kestrel.success_ratio`, `kestrel.denied_count`, `kestrel.incomplete_count`,
  and `kestrel.duration_ms` (with the legacy `kestrel.turn_duration_ms`
  alongside); the session summary aggregates them (`kestrel.turn_count` plus the
  same totals, `kestrel.duration_ms` alongside the legacy
  `kestrel.session_duration_ms`).

A `SessionStart` with `source` `compact`/`resume`/`fork` preserves the live
session (Claude Code reuses the `session_id`), so compaction never resets the
turn counter or duplicates turn ids.

Each hook invocation is its own process, so a tiny per-session state file
(`$KESTREL_OBS_CLAUDE_STATE_DIR`, else `$XDG_STATE_HOME/kestrel-obs-claude`, else
`$TMPDIR/kestrel-obs-claude/<session_id>.json`, written atomically) carries the
session/turn trace + span ids so spans across invocations share traces with no
daemon. The `openinference.project.name` (project = repo) is resolved from the
payload `cwd`'s git remote (`owner/repo`), else `$KESTREL_OTEL_PROJECT`, else
omitted, and cached per session. The script **always exits 0, prints nothing to
stdout** (Claude Code interprets `PreToolUse`/`Stop` stdout for gating), never
records the user prompt, and is an **instant no-op** — OpenTelemetry is never even
imported — when neither `OTEL_EXPORTER_OTLP_ENDPOINT` nor
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set.

### Wiring it into Claude Code

Hooks live in `~/.claude/hooks/` by convention; add a thin wrapper that pins the
endpoint and execs the console script (keeping the endpoint out of your global
env), matching the existing hook-directory layout:

```bash
# ~/.claude/hooks/obs-emit.sh
#!/usr/bin/env bash
exec env OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6006 kestrel-obs-claude-hook
```

```bash
chmod +x ~/.claude/hooks/obs-emit.sh
```

Then register the wrapper on the lifecycle events in `~/.claude/settings.json`.
`settings.json` supports **multiple** hooks per event, so these are **added
alongside** any existing entries (e.g. a `PreToolUse` guard) — never replace or
reorder them. Note `PostToolUseFailure` alongside `PostToolUse`: failed tools
fire a **separate** event, so without it errored tool calls would go unrecorded.
`PermissionDenied` is likewise its own event: without it a classifier-refused
tool is still recorded, but only as `incomplete` at turn end instead of `denied`:

```jsonc
{
  "hooks": {
    "SessionStart":       [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "UserPromptSubmit":   [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PreToolUse":         [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PostToolUse":        [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PostToolUseFailure": [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "PermissionDenied":   [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "Stop":               [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "SubagentStop":       [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }],
    "SessionEnd":         [{ "hooks": [{ "type": "command", "command": "~/.claude/hooks/obs-emit.sh" }] }]
  }
}
```

### Making the console script reachable

The script must be on `PATH` (or invoked by absolute path) from **any** cwd,
since Claude Code runs hooks from the project directory. Two options:

- **Host venv (absolute path).** If the host installs this package into a venv,
  point the wrapper at the absolute console-script path, e.g.
  `exec env OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6006 /opt/kestrel/venv/bin/kestrel-obs-claude-hook`.
- **`uv tool install`.** `uv tool install kestrel-feature-observability` puts
  `kestrel-obs-claude-hook` on the uv tools `PATH` (`~/.local/bin`). Note that a
  `uv tool` venv is **isolated**: editing this package's source (or bumping a
  dependency) does **not** update an already-installed tool — re-run
  `uv tool install --reinstall kestrel-feature-observability` (or, for local
  development, `uv tool install --editable .` and reinstall after dependency
  changes) to pick up changes.

## Privacy

The hook is observational — it never blocks, denies, or modifies. User-message content is **not** recorded (never stamped on any span); tool errors are truncated to 200 chars; exceptions in the hook are swallowed so they cannot affect agent operation.

Prompt capture is strictly opt-in: setting `KESTREL_OTEL_CAPTURE_PROMPTS=1` (off by default) stamps the turn's user prompt on the turn-root span as `input.value`, truncated to `KESTREL_OTEL_MAX_IO_CHARS` (default `20000`) characters — nothing changes unless an operator explicitly enables it at their own wiring point.

## Dependencies

- `kestrel-sovereign-sdk>=0.34,<1` — base `Feature`, `Hook`, and shared `metrics` module
- `httpx>=0.27.0` — lightweight HTTP client (OTLP/HTTP export transport)
- `opentelemetry-sdk` + `opentelemetry-exporter-otlp-proto-http` +
  `openinference-semantic-conventions` — the OTel span builders + OTLP export
- Optional `[metrics]` extra → `kestrel-sovereign-sdk[metrics]` → `prometheus-client`
- Optional `[fleet]` extra → `kestrel-sovereign-sdk>=0.34,<1` (the HostFeature
  contract for the Phoenix-embed console panel). No DB.

The base emitter has **no** runtime dependency on `kestrel-sovereign` (or any
fleet package); it emits OTel spans over OTLP/HTTP. The `[fleet]` extra adds only
the host SDK contract for the embed panel, so agents stay lightweight.

## Development

```bash
uv pip install -e '.[test]'
uv run pytest
```

## License

Apache-2.0
