Metadata-Version: 2.4
Name: cerberus-keys
Version: 0.3.0
Summary: Client library for Cerberus — per-key API abuse detection
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"

# Cerberus

Watches API-key usage metadata and tells you when one of your customers' keys
starts behaving unlike its own history — a shared, stolen, or scraped
credential — before the bill arrives.

Not bot detection. Specifically: *this paying customer's key is doing
something it has never done, and it's costing you money.*

---

## What Cerberus receives

**Twelve fields. Nothing else.** The ingest endpoint rejects any unknown field
by name, so this list is enforced in code rather than promised in prose.

| Field | Meaning |
|---|---|
| `ts` | ISO 8601 UTC timestamp |
| `key_fp` | Fingerprint of your API key ID — never the key |
| `endpoint` | Route **template** (see below) |
| `tokens_in` / `tokens_out` | Token counts |
| `latency_ms` | Request latency |
| `status` | HTTP status code |
| `ip_fp` | Fingerprint of the client IP — never the address |
| `ip_net_fp` / `ip_block_fp` | Fingerprints of the surrounding network |
| `ip_family` | `v4` or `v6` |
| `cost` | Optional, in USD |

**No content, ever.** No prompts, no responses — not hashed, not truncated,
not optional.

**Thirty-day rolling retention.**

## Why we cannot see your keys or your users' IP addresses

Fingerprints are HMAC-SHA256 under a secret salt **you generate and we never
receive**, truncated to 128 bits.

This matters because a plain hash is not anonymisation: the IPv4 space is
2^32, so a GPU builds the whole rainbow table in minutes. With a salt we never
hold, we cannot reverse a fingerprint even in principle — and because the salt
is per-tenant, the same IP hitting two of our customers produces two unrelated
fingerprints. We could not build a cross-customer profile if we wanted to.

### Salt handling

- Store it wherever your other secrets live, **not in a config file**.
- Use the **same salt on every instance**, or one key looks like several.
- **Do not rotate it.** Every historical fingerprint becomes uncorrelatable
  and all baselines reset.
- If you lose it, baselines reset. Nothing else breaks.

### Looking up a fingerprint at 2am

An alert names a key by fingerprint, because that is all we have. You can
always map it back — you hold both the salt and your key list:

```python
from cerberus_keys import fingerprint
{fingerprint(k, salt): k for k in my_api_keys}
```

We cannot do this lookup. That is the point. Build the index once during
onboarding and keep it.

## Installing

```bash
pip install cerberus-keys
```

Zero runtime dependencies, standard library only.

```python
from cerberus_keys import Cerberus

client = Cerberus(ingest_token, salt, endpoint_url="https://.../v1/events")
client.record(api_key=key, ip=client_ip, endpoint="/v1/chat",
          tokens_in=n_in, tokens_out=n_out, latency_ms=ms, status=200)
```

### Is it actually recording?

```python
client.health
# {"ok": True, "dropped": 0, "rejected": 0, "invalid": 0,
#  "queued": 0, "last_rejection": None}
```

`ok` is the one thing worth putting on your own status page. It goes **False**
when the client is discarding data for a reason that will not fix itself:

- `rejected` — the server refused a batch permanently (bad token, malformed
  field, an event past the backfill window). Retrying cannot help.
- `invalid` — the client could not even build the event. **The usual cause is
  `ip` not being a bare address**: a port, an `unknown`, an RFC 7239 `for=`
  value or a bracketed IPv6 will all do it. See the gateway section above.

`dropped` is different and is **not** an error: it means the buffer was full
and the client shed load rather than slowing your request path. `queued`
includes any batch in flight, so `dropped + rejected + invalid + queued`
accounts for everything you recorded.

`record()` **never blocks, never raises, and never adds latency** to your
request path. It enqueues and returns; a full buffer drops rather than
applying backpressure; every exception is swallowed at the boundary. A
monitoring library that can stall the thing it monitors is not worth running.
Dropped events are visible on `client.dropped` and are never an error.

## If you run a gateway, proxy or load balancer, read this first

**`ip` must be the real client address, not the address of your own
infrastructure.** This is the one misconfiguration that leaves Cerberus
installed, reporting healthy, and unable to ever fire.

Behind any load balancer, ingress or CDN, the socket peer your framework
reports (`request.client.host`, `req.socket.remoteAddress`, `REMOTE_ADDR`) is
*the load balancer*, identically on every request. Every key then has exactly
one distinct address forever, the fan-out rule compares one against one, and
no leak can produce a deviation. Events arrive, `/v1/status` says the key is
watched, and nothing can ever alert.

Take the leftmost entry of `X-Forwarded-For`, which is the original client:

```python
from cerberus_keys import client_ip_from_forwarded_for

ip = (client_ip_from_forwarded_for(request.headers.get("x-forwarded-for", ""))
      or request.client.host)
client.record(api_key=key, ip=ip, endpoint=route_template, ...)
```

Send the leftmost entry only. The header is a chain (`client, proxy1, proxy2`)
and fingerprinting the whole string counts every hop combination as a separate
address, which makes fan-out counts meaningless in the other direction.

**On LiteLLM this is a config flag, not code** — see the section below; it is
the second of the two required blocks and the one that is easy to miss.

**How to check you got it right.** `GET /v1/status` reports what Cerberus can
see per key. A key sitting at one distinct address is reported as below the
detection floor. Cerberus also checks each account for the whole-fleet version
of this shape and tells you unprompted once enough keys share one address —
but that notice has a threshold, so on a small account the `/v1/status` reading
is the check that matters. Don't rely on either; set the header.

## You will know within the hour that it works

Once your first events reach us, Cerberus posts a one-time confirmation to your
Slack webhook. It arrives on the next hourly pass — so within the hour, not
instantly — and it tells you:

- how many events we received, and how many distinct API keys they came from
- how many distinct client IPs we can see
- when detection starts, since we need about a week of history per key first

**If it does not arrive, something is wrong, and that is the point.** Cerberus
can fail in a way that looks exactly like working: this SDK drops failed sends
silently, on purpose, so it can never add latency or raise into your request
path. A Cerberus that is receiving nothing looks identical to a Cerberus that
is watching quietly. The confirmation exists so that silence means something
specific instead of nothing.

If an hour passes and no message arrives, check in this order:

1. **Is a Slack webhook configured on your account?** No webhook, no message —
   and no alerts either.
2. **Is the ingest token right, and not revoked?** A bad token gets a 401,
   which the SDK drops.
3. **Is `endpoint_url` correct?** Anything non-2xx is dropped the same way.
4. **Is your process alive long enough to flush?** The client batches. A script
   that exits immediately should call `.close()`.
5. **Is `ip` a bare address?** Behind a proxy it often is not — a port, an
   `unknown`, an RFC 7239 `for=` value. Every such event is discarded before it
   reaches the network.

**`client.health["ok"]` is the fastest local check**, and it is False for
exactly the two causes that never fix themselves: `rejected` (the server
refused) and `invalid` (the client could not build the event). Reading
`dropped` alone will not tell you — a full buffer is healthy shedding, and
until 0.3.0 a malformed address was counted there too, so a gateway could
discard every event while `ok` stayed True.

### `endpoint` must be a route template

Send `/v1/orgs/{org_id}/chat`, **not** `/v1/orgs/acme-corp/chat`. Live paths
routinely carry identifiers, and this field would carry them to us.

The ingest endpoint rejects paths containing `@`, UUIDs, or long digit and hex
runs — but that is defence in depth, not a guarantee. `/v1/orgs/acme-corp/chat`
defeats every one of those checks. **The template requirement is the
mechanism; we cannot detect every violation of it.**

## Importing history so detection starts today

Cerberus will not judge a key until it has enough of that key's own history:
**500 requests across 40 active hours from 3 or more addresses, inside a
rolling 7 days.** Live traffic takes about a week to get there. If you already
have request logs, import them and the baseline closes immediately — the
import is rolled into hourly aggregates synchronously, so the keys are watched
the moment it finishes.

Pass `backfill=True` and use a **separate client** for the import:

```python
importer = Cerberus(
    ingest_token, salt,
    endpoint_url="https://api.cerberushq.dev/v1/events",
    backfill=True,
)

for row in historical_rows:          # oldest first
    importer.record(
        api_key=row.api_key, ip=row.client_ip, endpoint=row.route_template,
        tokens_in=row.tokens_in, tokens_out=row.tokens_out,
        latency_ms=row.latency_ms, status=row.status, cost=row.cost,
        ts=row.timestamp,            # timezone-aware, historical
    )

importer.flush(timeout=300)
importer.close(timeout=60)

# ALL FOUR COUNTERS. `rejected` counts only batches the server refused
# permanently, which is the one way a real import does not fail -- a dropped
# connection or a killed process shows up in `dropped` or `queued`, and an
# event the client could not build at all shows up in `invalid`. Asserting on
# `rejected` alone passes on every realistic half-landed import.
h = importer.health
assert (h["rejected"] == 0 and h["dropped"] == 0
        and h["queued"] == 0 and h["invalid"] == 0), (
    f"partial import, do not trust the baseline: {h}")
```

A half-landed import is worse than none: the keys still become "watched" and
nothing reports that the baseline is thin. Re-running the whole import is safe
— rolling an hour recomputes it from events rather than adding to it.

**Import 8 days, not 30.** Only the trailing 7 days form the baseline. Older
events are accepted and stored — up to 90 days, matching how long aggregates
are kept — but they sit outside the window and change nothing about detection.
Importing a month costs you the upload and buys nothing over importing a week.

**Ordinary batches are refused past 48 hours.** That is what the flag changes,
and it is why passing historical timestamps without it fails: every batch comes
back `400` naming the fix. Check `health["rejected"]` when the import finishes
— a backfill that half-lands leaves a baseline computed from partial history,
which is worse than no baseline at all.

**Do not reuse the importer for live traffic.** Every batch it sends carries
the flag, which suppresses the lateness marking live events rely on to be
re-rolled. One instance is an importer or it is not.

**An import never pages you.** The hours inside it are not evaluated, so
nothing in your history can raise an alert — a month-long import would
otherwise page you once for every incident in it. The exception is the most
recent closed hour, which the next hourly pass evaluates like any other.

**It inherits whatever was happening.** If a key was already being shared
during the window you import, that becomes its normal and Cerberus will not
flag it until it gets worse still. This is not a property of importing —
waiting seven days does the same thing, because those days also contain the
abuse — it is a property of learning a baseline from history at all.

## If you run LiteLLM

This is the cheapest integration on offer: LiteLLM's callback payload already
carries eight of the twelve fields, so there is no instrumentation to write.
It is **two config blocks, and both are required.**

First, a file the proxy can import. Put it next to your config:

```python
# cerberus_callback.py
import os
from cerberus_keys.litellm import CerberusLogger

cerberus = CerberusLogger(
    ingest_token=os.environ["CERBERUS_INGEST_TOKEN"],
    salt=bytes.fromhex(os.environ["CERBERUS_SALT"]),
    # Note the `api.` subdomain; this is the endpoint_url from your signup
    # response. The apex is the website and does not accept events.
    endpoint_url="https://api.cerberushq.dev/v1/events",
)
```

**Block one** registers it:

```yaml
litellm_settings:
  callbacks: cerberus_callback.cerberus
```

**Block two** is the one that matters, and it is in a *different* section:

```yaml
general_settings:
  use_x_forwarded_for: true   # REQUIRED
```

Without it, LiteLLM reports `request.client.host` — which behind any load
balancer, ingress, or CDN is *the load balancer's address on every request*.
Distinct-IP counts collapse to one permanently and fan-out detection silently
cannot fire, while looking exactly like a clean bill of health. It is the only
way to install Cerberus, have it appear to work, and get nothing from it.

Cerberus checks for this shape and tells you if it sees it. Don't rely on
that; set the flag.

### Two things that are different on LiteLLM

**Resolving a fingerprint takes the hash, not the key.** LiteLLM never hands a
callback your raw key — it passes `user_api_key_hash`, which is
`sha256(key).hexdigest()`. So the fingerprint in an alert is
`HMAC(salt, sha256hex(key))`. Calling `resolve()` with raw keys matches
nothing, silently, and reads exactly like "that key isn't ours" at the worst
possible moment. Map them first:

```python
from cerberus_keys import resolve
from cerberus_keys.litellm import key_fingerprint_input

resolve(fp, [key_fingerprint_input(k) for k in your_keys], salt)
```

**Streaming latency is time-to-first-token.** LiteLLM's `response_time` is
`completion_start_time - start_time` when `stream=True`, not the full
duration. Latency is a digest signal rather than a fan-out condition, so this
does not affect what pages you — but a streaming-heavy proxy will show lower
latencies than its users experience, and that is a property of the source, not
of Cerberus.

The proxy path only. `requester_ip_address` is populated in LiteLLM's
proxy-side request handling, so the LiteLLM **SDK used as a library** does not
carry it — take the normal `Cerberus(...).record(...)` path there.

## What it detects

**One real-time signal: key fan-out.** Not "many IPs" — *many IPs each doing
very little*, which is the shape of a shared credential and what separates it
from your infrastructure scaling up. Scaling three containers to forty raises
volume in proportion; a leaked key inverts that. Five conditions must all
hold, sustained across consecutive hours, before anything pages you.

Everything else — cost and volume anomalies — goes in a daily digest. A
customer who just launched and 10x'd their usage looks identical to abuse, and
that customer is the best thing that happened to you this quarter.

## Known limits, stated plainly

- **Keys used from fewer than 3 distinct IPs are not protected at all.** A
  key that lives on one or two servers never builds a baseline the rule can
  compare against, so a leak of that key -- however dispersed, however
  sustained -- does not fire. This is the single biggest gap, it covers the
  most common key shape, and `GET /v1/status` names the affected keys as
  `below_detection_floor` rather than pretending they are still warming up.
  The floor is a calibration threshold; it will move only on replay evidence,
  not by guess.
- **Keys already spread across many IPs are under-protected.** The rule is
  relative to each key's own history, so a key that normally lives on 200 IPs
  needs a far larger jump to trip. Conservative by design; you should know it.
- **A leak that ramps slowly enough is never caught.** The baseline tracks it
  upward and nothing fires. Detecting that needs a long-horizon reference we
  don't yet have.
- **Concentrated datacenter scrapers don't page.** A scraper rotating inside
  one provider's range looks concentrated, and the rule requires dispersion.
  Deliberate: a missed scraper is cheaper than a false page.
- **A serverless migration can page you.** If you move a key from a few
  servers to many small workers (a serverless or autoscaling rollout) at
  roughly constant traffic, and those workers are spread across many network
  blocks — which cloud egress usually is — that has the same shape as a
  leaked key being used from many places, and the rule can fire. It is the one
  false alarm the design cannot rule out from the data it holds: it cannot
  tell your cloud's address ranges apart from a stranger's. If it happens,
  acknowledge the alert (it offers a "this key is legitimately distributed"
  link) and it stops. Tell us if you have a rollout planned and we can quiet
  the key ahead of time.
