Metadata-Version: 2.5
Name: devoriz
Version: 0.2.0
Summary: Read-only production triage: digest your logs, model your flows, and give an agent a safe way to investigate.
Project-URL: Homepage, https://github.com/ameerfayiz/devoriz
Project-URL: Documentation, https://github.com/ameerfayiz/devoriz/tree/main/docs
Project-URL: Issues, https://github.com/ameerfayiz/devoriz/issues
Project-URL: Changelog, https://github.com/ameerfayiz/devoriz/blob/main/CHANGELOG.md
Project-URL: Source, https://github.com/ameerfayiz/devoriz
License: Apache-2.0
License-File: LICENSE
Keywords: agent,elasticsearch,grafana,incident-response,kafka,llm,loki,observability,pii,redaction,sre,triage
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: pyyaml<8,>=6
Provides-Extra: agent
Requires-Dist: anthropic<1,>=0.79; extra == 'agent'
Requires-Dist: cryptography<51,>=42; extra == 'agent'
Requires-Dist: fastapi<1,>=0.110; extra == 'agent'
Requires-Dist: pydantic<3,>=2; extra == 'agent'
Requires-Dist: ruamel-yaml<0.20,>=0.18; extra == 'agent'
Requires-Dist: uvicorn<1,>=0.27; extra == 'agent'
Provides-Extra: all
Requires-Dist: anthropic<1,>=0.79; extra == 'all'
Requires-Dist: cryptography<51,>=42; extra == 'all'
Requires-Dist: fastapi<1,>=0.110; extra == 'all'
Requires-Dist: mcp<2,>=1.0; extra == 'all'
Requires-Dist: pydantic<3,>=2; extra == 'all'
Requires-Dist: pypdf<7,>=4; extra == 'all'
Requires-Dist: ruamel-yaml<0.20,>=0.18; extra == 'all'
Requires-Dist: uvicorn<1,>=0.27; extra == 'all'
Provides-Extra: dev
Requires-Dist: coverage[toml]>=7; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.0; extra == 'mcp'
Provides-Extra: pdf
Requires-Dist: pypdf<7,>=4; extra == 'pdf'
Provides-Extra: share
Requires-Dist: cryptography<51,>=42; extra == 'share'
Description-Content-Type: text/markdown

# Devoriz

[![CI](https://github.com/ameerfayiz/devoriz/actions/workflows/ci.yml/badge.svg)](https://github.com/ameerfayiz/devoriz/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/devoriz.svg)](https://pypi.org/project/devoriz/)
[![Python](https://img.shields.io/pypi/pyversions/devoriz.svg)](https://pypi.org/project/devoriz/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

Read-only production triage: digest your logs, model your flows, and give an
agent a safe way to investigate.

```bash
pip install devoriz
devoriz init myapp                         # scaffold a profile for your platform
devoriz doctor                             # check it, without querying anything
devoriz inventory && devoriz map           # discover environments and targets
devoriz logs errors -e prod -t api -s 2h   # a digest, not a wall of lines
devoriz ui                                 # the same tools, driven by an agent
```

Or skip the first four: `pip install 'devoriz[agent]' && devoriz ui` opens a
settings screen that creates the profile, fills in the environments and
credentials, edits the redaction policy and runs the same checks — and a **Docs**
screen with every guide in it, starting from
[getting-started](docs/getting-started.md).

`diz` is installed as a shorter alias for the same command — the docs use
`devoriz` throughout, but nobody types seven characters at 3am.

Point it at Grafana/Loki, Elasticsearch and Kafka — or at whatever you run, via a
plugin — and describe your platform once in a **profile**. Everything after that
is generic.

---

## Why this exists

Four problems, each of which has a cheap answer that almost nobody implements:

**Logs are unreadable at incident volume.** 50,000 lines is not a thing a human
or a model can read. Every command here defaults to a *fingerprint digest*:
lines collapsed by message template, with counts, a window and one exemplar. A
digest of 50,000 lines costs about what a digest of 200 costs, so you start with
the shape of the problem and drill into exactly one template.

```
 count  window        service                  template
  4812  13:20→14:05   api                      failed to resolve customer: <NUM>
    31  13:22→13:24   workers                  Cannot read properties of undefined (reading '<WORD>')
```

**"Where did it stop?" is a different question from "what is erroring?"** A
digest cannot answer it. `flow` is an executable model of how one entity moves
through your platform — the stages, the cheapest observable that proves each one,
the lag budget between them — and `probe` walks it, anchoring each query on the
*previous* stage's last line. Seven stages cost about fifteen lines of output
whether the services logged 200 lines that hour or 200,000.

```
✓ checkout       08-11 17:38:05.800           20 lines  api
✓ payment        08-11 17:38:06.848 +1s       13 lines  workers
✗ confirmation   nothing in 17:38:06->17:53:06  workers  [optional]

STOPPED after payment (last seen 17:38:06.848) — confirmation never saw it within 15m
  if: payment succeeded but no email
      look for: the template id, or a bounce in the provider webhook log
```

The model corrects itself: a stage that was seen climbs `assumed → likely →
confirmed`, a probe that worked is tried first next time, and a stage that missed
while a *later* one hit is marked `refuted` rather than quietly believed.

**Investigations forget everything.** Notes go stale silently, so nobody trusts
them, so nobody writes them. Here a note is a *belief* with a standing —
`confirmed` > `likely` > `assumed`, `~` when nobody has re-checked it in weeks,
`refuted` when it was disproved — and settling one is a single cheap call at the
end of a turn. Refuted notes are kept, because "we checked, it is not that" is
the cheapest thing memory can say.

**You cannot paste production logs into a model.** Not if they contain customers.
Every identifier is replaced by a **keyed pseudonym** before it reaches an LLM, a
terminal or a ticket:

```
9876543210          ->  <PHONE:7QK3M2>         9864563210  ->  <PHONE:X4BN0R>
+91 98765 43210     ->  <PHONE:7QK3M2>         same number, same token
jane.doe@gmail.com  ->  <EMAIL:J5J8QP>
ops@example.com     ->  <EMAIL:07QWK4>@example.com    staff domain survives
```

A partial mask (`98****210`) makes those two numbers indistinguishable and
destroys the most useful join in an investigation. `<phone>` is worse. A
pseudonym is **distinguishing** (different values, different tokens),
**correlating** (same value, same token, in every backend and every future
session) and **one-way** (HMAC-SHA256 under a key that never leaves the machine).

---

## Read-only by construction

Not by convention, and not by asking the model nicely:

- The HTTP layer refuses any method but GET unless the caller is a login handler,
  so producing a Kafka message or resetting an offset is unreachable code.
- Elasticsearch paths are validated per **segment** against a read allowlist, so
  `logs-x/_search/../_bulk` is refused rather than forwarded.
- The agent has no shell, no `kubectl`, and no write tool. The only things it can
  change are its own notes and its own flow models, both through a schema.
- `read_file` and `search_code` are confined to the configured repo roots by
  canonical-path check.
- `--no-mask` exists for an engineer at a terminal, relaxes only to
  *secrets-only* (no flag prints a bearer token), and is refused to the agent
  outright.
- **A pseudonym may never be searched for.** The token is minted here; every
  backend stores the real value, so searching for one returns zero rows — and a
  well-formed query returning zero rows reads as *the service never saw them*.
  Every search term is checked before it leaves.

---

## Configure it for your infra

One directory describes one platform. Switch platforms with `--profile`.

```
~/.devoriz/
  devoriz.yaml                   default profile, shared settings, plugins
  plugins/*.py                   drop-in plugins, no packaging required
  profiles/myapp/
    profile.yaml                 environments, credentials as ${VAR}, discovery rules, repos
    .env                         the secrets those vars resolve to (chmod 600)
    prompt.md                    what the agent needs to know about your platform
    pii.yaml                     what counts as identifying here
    skills/*.md                  how *you* triage a specific situation
    flows/*.yaml                 executable models, grown by probing
    memory/*.md                  what past investigations proved
    .state/                      generated: targets, inventory, cookies, history, keys
```

`profile.yaml` never holds a secret — it names them:

```yaml
environments:
  prod:
    loki:
      url: https://grafana.example.com
      user: ${GRAFANA_USER}
      password: ${GRAFANA_PASSWORD}
    kafka:
      url: https://kafka-ui.example.com
      user: ${KAFKA_UI_USER}
      password: ${KAFKA_UI_PASSWORD}
```

Targets are discovered rather than typed: `devoriz inventory` sweeps every
datasource once and caches the label schema; `devoriz map` proposes environments
and named targets from it and asks you to confirm. Where your naming is unusual,
`discovery:` rules teach the mapper — regions, variants, how pod names collapse
into components, which host prefix is which tier. Hand-written entries in
`profile.yaml` always win over generated ones, so a fix is permanent.

`devoriz doctor` reports what is configured, what is missing and what would be
tried — without running a single query.

None of this needs an editor. `devoriz ui` → **Settings** is a form over the same
files: profiles (create, copy, switch, set the default, remove from the list
without deleting anything), every environment and backend block, the credentials
behind each `${VAR}`, the agent's prompt and skills, MCP servers, and the
redaction policy with a live tester. Every write is validated before it lands and
backed up after, and the raw YAML is always one click away.

---

## Extend it

Everything is a plugin, including the parts that ship in the box. `devoriz.backends`
and `devoriz.cli.builtin` register themselves through the same public API a
third party uses, which is how the API stays honest.

A plugin is any module with `register(registry)`, found three ways: a `.py` file
in `<workspace>/plugins/`, a module named in `plugins:` in your profile, or a
`devoriz.plugins` entry point in an installed package.

```python
from devoriz.plugins import BackendSpec

def register(reg):
    reg.add_backend(BackendSpec(
        name="datadog", summary="Datadog Logs", section="datadog",
        factory=lambda cfg, target=None, verbose=False: DatadogSource(cfg),
        add_parser=build_parser, probeable=True,
        agent_help="Datadog Logs. Subcommands: errors, search, trace.",
    ))
```

That one registration adds `devoriz datadog …` to the CLI, makes Datadog-backed
targets probeable by `flow`, and tells the agent the command exists — because the
tool list and the system prompt are both assembled from the registry rather than
written down. Six kinds of contribution:

| Kind | Adds |
|---|---|
| `BackendSpec` | a data source: a CLI command, a `flow` probe target, an agent tool value |
| `CommandSpec` | a CLI subcommand that is not a data source |
| `AgentToolSpec` | a typed tool the model calls directly |
| `SkillSpec` | a procedure in markdown — **no Python at all** |
| `DecoderSpec` | how to parse your house log format |
| `DetectorSpec` | an identifier shape the redactor should recognise |

**Skills are the cheapest extension point.** A markdown file with a `when:` line
in `<profile>/skills/` encodes how your team triages one situation. Always-on
skills are inlined in the prompt; the rest are listed by trigger and pulled with
`load_skill` when they apply, so a hundred skills stay affordable.

```markdown
---
name: stuck-consumer
when: a queue is not draining, or lag is climbing
---
1. `devoriz kafka lag -e <env>` — STABLE with climbing lag means stuck, not crash-looping…
```

See [docs/plugins.md](docs/plugins.md), [docs/skills.md](docs/skills.md) and
[examples/plugins/](examples/plugins/).

---

## The agent

`devoriz ui` opens a chat UI backed by an agent that drives the same commands you
do, asks clarifying questions when the answer depends on them, and writes down
what it learns.

A message can carry more than words. Drop a file on the window, paste a
screenshot, or use ＋: a text file is inlined redacted — head first, with
`read_attachment` to page or grep the rest — while an image goes as it is, which
the composer warns you about because nothing can pseudonymise the inside of a
screenshot. A **document** — PDF, `.docx`, `.pptx`, `.xlsx`, OpenDocument — is
converted to text on your own machine first, so the incident report full of
customers goes through the same gate every log line does. (Office formats need
nothing installed; PDF uses `pypdf` or `pdftotext`, and a scan neither can read
is handed over whole, flagged as unredacted.) And `@` completes against your
profile: `@api` becomes
`repo="api"` for the code tools, `@prod` becomes `-e prod` for every query,
aliases included, so the agent never spends a turn asking which environment you
meant.

No API key required: it uses the OAuth token Claude Code already stored. That
token shares a rate limit with your interactive session, so set
`ANTHROPIC_API_KEY` for an independent quota. MCP servers (Jira, GitHub, …)
configured in `<profile>/mcp.json` become tools with the same redaction gate over
their results — a ticket describing a customer's problem carries identifiers no
log rule would ever see.

Recall and persistence are one loop:

```
1. RECALL      memory_search → index lines ranked by how well proved
2. INVESTIGATE logs · kafka · docs · search_code · flow probe
3. SETTLE      memory_observe confirmed → climbs, clock reset
                              refuted   → dead, kept, unranked
4. WRITE       memory_write / flow_write — the corrected claim
5. ANSWER      a bounded persist follow-up fires only if the turn owed memory something
```

---

## Install

```bash
pip install devoriz            # the CLI: PyYAML only
pip install 'devoriz[all]'     # plus the agent, its web UI and profile sharing
```

The extras are opt-in, so `pip install devoriz` really does install nothing but
PyYAML — `devoriz[agent]` is what brings FastAPI and uvicorn for `devoriz ui`, and
`devoriz[share]` the cipher for `devoriz share`.

Python 3.10+. No services to run, no database, no daemon.

## Tests

```bash
tests/run.sh           # offline: no network, no credentials, no API quota
devoriz pii self-test  # the redaction corpus against the policy you installed
```

Each suite runs in its own process, because they install different workspaces and
different redaction policies. `pytest tests/t_plugins.py` works for a single one.

## Documentation

Every guide below also ships **inside the package**: `devoriz ui` → **Docs**
renders them, searches them, and links between them — so a pip install carries
its own manual.

| | |
|---|---|
| [docs/getting-started.md](docs/getting-started.md) | zero to a shared team profile, in eleven stages |
| [docs/configuration.md](docs/configuration.md) | profiles, environments, targets, discovery rules |
| [docs/settings.md](docs/settings.md) | configuring all of it from the browser |
| [docs/sharing.md](docs/sharing.md) | sending a profile to a colleague as one sealed file |
| [docs/plugins.md](docs/plugins.md) | adding a backend, a command, a tool |
| [docs/skills.md](docs/skills.md) | teaching it how your team triages |
| [docs/flows.md](docs/flows.md) | building an executable model of a journey |
| [docs/pii.md](docs/pii.md) | the redaction policy, and what it does not claim |
| [docs/architecture.md](docs/architecture.md) | ports, adapters, and where to put a change |
| [docs/migrating.md](docs/migrating.md) | porting a single-platform fork onto profiles |

## Contributing

Issues and pull requests are welcome. The full guide is in
[CONTRIBUTING.md](CONTRIBUTING.md); the short version:

```bash
git clone https://github.com/ameerfayiz/devoriz && cd devoriz
python -m venv .venv && source .venv/bin/activate
pip install -e ".[agent,dev]"

tests/run.sh                 # must be green
ruff check src tests         # must be clean — CI fails on any violation
```

**The cheapest contribution is not Python.** A skill is a markdown file
describing how you triage one situation, and a detector is an identifier shape —
both are useful to everyone and neither needs a code change. See
[docs/skills.md](docs/skills.md) and [docs/pii.md](docs/pii.md).

Two things in this codebase are load-bearing rather than stylistic, and a change
that touches either needs to say so in the pull request:

- **Nothing writes.** Non-GET is unreachable from the query path. If a change
  needs a POST to a backend, open an issue before a patch.
- **Nothing identifying escapes.** Every path to a model, a terminal, a file or a
  ticket goes through `scrub()`. A new detector comes with two corpus cases in
  `t_pii.py` — one that must be redacted, and one that must survive, because
  over-redaction destroys the joins an investigation runs on.

CI runs the suite on Python 3.10–3.13, installs the built wheel and drives it end
to end, and fails if a `.env` or `.state/` path is ever committed.

Found a security issue — a redaction bypass, a way to make it write, a
confinement escape? **Do not open an issue.**
[Report it privately](https://github.com/ameerfayiz/devoriz/security/advisories/new);
see [SECURITY.md](SECURITY.md) for what is in scope and what to include. By
participating you agree to the [Code of Conduct](CODE_OF_CONDUCT.md).

## License

Apache-2.0.
