# stapel-alerts 0.2.3

An alert store for a fleet that has no Sentry, and the same capture() call whether one is connected or not. One row per BUG (fingerprint, status, count, first/last seen, the release that claimed the fix, whether it came back) rather than a search over log text — because the question a fix wave is reconciled against is 'which bugs are open, which are fixed, and which regressed', and that needs a status column. Every input is wired by installing the app: a WARNING+ log handler, the DRF 5xx hook, Celery task failures, core's DLQ parks, and the comm handler exceptions that are RETURNED rather than raised and which no except clause anywhere would otherwise see. Monoliths write in process; microservices POST with a per-service key, buffered and retried, and fall through to Telegram when the owner is unreachable — never over the bus, because the bus is the thing whose failures this records. A Prometheus blind-spot watchdog files the monitoring stack's own gaps as issues, on the principle that 'no alerts firing' is the observable state of both a healthy fleet and a blind one.

Contract: axes 6 · surface 18 · extension points 7 · operations 5 · error codes 49.
Generated from docs/capabilities.json by `stapel-llms-txt` — do not edit; drift-gated by `make contract-check`.

## Configuration axes — what a product switches on
Settings keys; `default` is what you get by saying nothing. Turning an axis off unmounts the operations it gates.
- CAPTURE_5XX [bool, default true] — Collect server errors
  Whether anything the fleet exception handler answers with a 5xx becomes an issue. On by default. A 400 is never captured at any setting — a validation error is the API working, and a store that files them is a log nobody reads.
- CAPTURE_CELERY [bool, default true] — Collect failed background jobs
  Whether a Celery task that failed after its retries becomes an issue. On by default, and a no-op when celery is not installed. A failed task is work that did not happen and that nobody is waiting on — the class of failure most likely to be silent.
- CAPTURE_DLQ [bool, default true] — Collect dropped work
  Whether a bus dead-letter park or a task-ledger unprocessable record becomes an issue. On by default. Work the system gave up on is an alert BY CONSTRUCTION, never a judgement call: the DLQ counter says how much was dropped, this says what, with the traceback.
- MODE [enum, default "auto"] — Where the alerts live
  Whether this process OWNS the alert store or reports into somebody else's. "auto" (the default) reads the presence of OWNER_URL, so a monolith configures nothing and a microservice configures exactly the two settings it obviously must have anyway. "owner" / "reporter" force it. This is the only axis that changes what the module IS rather than what it collects. Values: auto, owner, reporter.
- MONITORING [enum, default {"PROMETHEUS_URL": "", "TARGETS": [], "METRICS": [], "HEARTBEAT_ALERT": "", "ALERTMANAGER_URL": "", "SERVICE": "", "TIMEOUT_SECONDS": 5.0, "SCHEDULED": false}] — Watch the monitoring
  The Prometheus blind-spot watchdog. Off until PROMETHEUS_URL is set, which is the right default for a library — a watchdog pointed at nothing reads exactly like a fleet with nothing wrong. Configured, it asks every five minutes whether the monitoring stack can still see: up == 0 on each scrape target, a target with no up series at all, absent() on named metrics, a dead-man's-switch alert that stopped firing, and Alertmanager's active silences. Each blind spot becomes an issue in this same tracker AND a message on the fallback channel, because the normal notification thresholds would be silent on the ninth consecutive run — which is not less urgent than the first.
- NOTIFY [enum, default "telegram"] — What wakes somebody up
  The channel used when the alert store itself cannot be reached, and by the monitoring watchdog. "telegram" (the default) prefers stapel-notifications' channel when that is configured and falls back to a direct Bot API call; "none" disables it; a dotted path or a callable plugs in whatever the deployment actually reads. The direct path exists because in a small fleet the notifications module lives IN the alerts owner, and a fallback that needs the thing that is down is not a fallback.

## Usage surface — call these before writing your own
This is the answer to "does Stapel already have something for X?". `instead of` names the outside symbol this one displaces.
### predicate
- check_alertmanager — stapel_alerts.monitoring.check_alertmanager
  Alertmanager's active silences, and an Alertmanager that does not answer. A silence is somebody muting a page at 3am; it becomes a blind spot the moment they go back to sleep, and the thing nobody does is come back and remove it — which is why it is reported as an issue with a rising count rather than left to a UI nobody opens.
- check_heartbeat — stapel_alerts.monitoring.check_heartbeat
  The dead-man's switch: is the always-firing HEARTBEAT_ALERT still firing? If it is not, the whole chain — rule evaluation, Alertmanager, the receiver — is broken, and every other alert in it is broken the same way while showing nothing.
- check_metrics — stapel_alerts.monitoring.check_metrics
  absent(<metric>) on each named metric. absent() is the only expression that fires when a metric stops existing; every other one returns nothing, and nothing does not fire.
- check_targets — stapel_alerts.monitoring.check_targets
  up == 0 on each configured scrape target, and the harder case: a target with no up series at all, which means it was dropped from the scrape config or renamed. That second one is strictly worse than a down exporter — every expression over its metrics returns no data, and an alert on no data does not fire.
- collect — stapel_alerts.monitoring.collect
  Ask every configured monitoring question once and return the findings, without filing, closing or notifying anything. This is what `manage.py alerts_watch_monitoring --dry-run` uses — the way to find out what the watchdog would say before it says it to somebody's phone at 3am.
- expected_checks — stapel_alerts.monitoring.expected_checks
  Every (check, target) pair this configuration CAN produce. Recovery reads it: a finding's absence is only evidence of recovery if the run actually asked the question, so removing a target from the config must not close its issue as 'recovered' — a lie the tracker would then repeat forever.
- fingerprint_of — stapel_alerts.monitoring.fingerprint_of
  The fingerprint capture() will compute for a given Finding — the same expression the ingest makes, written once so recovery can find an issue by its id instead of by matching text. If you need to correlate a monitoring issue with the check that produced it, this is the link.
- instant_query — stapel_alerts.monitoring.instant_query
  Run one PromQL instant query against the configured Prometheus and return its result vector. Raises PrometheusUnreachable for a transport failure, a non-2xx, OR a 200 whose body says status != success — all three mean 'this question was not answered', and returning an empty vector instead would make every check above report health at exactly the moment nothing could be seen.
- is_configured — stapel_alerts.monitoring.is_configured
  Has a host pointed the watchdog at a Prometheus? Read this before scheduling it — unconfigured is a no-op rather than an error, and the failure mode of an unconfigured watchdog is that it looks exactly like a fleet with nothing wrong.
- monitoring_settings — stapel_alerts.monitoring.monitoring_settings
  The STAPEL_ALERTS["MONITORING"] block as a plain dict, read at call time. Use it instead of reaching into django.conf.settings: the block resolves through AppSettings (dict, then flat setting, then environment, then default), and a host that indexes the raw setting sees only the first of those four.
### factory
- alerts_watch_monitoring — stapel_alerts.beat.alerts_watch_monitoring
  One watchdog pass as a plain callable, registered as a Celery task when celery is installed. This is the name the shipped beat entry references; call it directly from a host scheduler that is neither celery beat nor cron.
- announce — stapel_alerts.monitoring.announce
  Push the run's verdict at the NOTIFY seam directly, on EVERY run that has something to say. Deliberately not subject to the new/regressed/spike thresholds the rest of the module uses: a blind spot on its ninth consecutive run is not less urgent than on its first, and 'the monitoring is still blind' is the message somebody needs at the moment they look at their phone.
- capture — stapel_alerts.capture.capture
  instead of: logging.exception, sentry_sdk.capture_exception, stapel_alerts.models.ErrorEvent.objects.create
  The one call every input and every library makes: capture(exc) or capture("text", level=..., context={...}). Where it goes is a deployment question this answers, not a caller question — the local store when this process owns it, HTTP-with-a-service-key when it does not — which is what makes the same line correct in a monolith and in a microservice. It cannot raise (every caller is on a failure path) and it is rate limited per fingerprint, with what the limiter dropped carried into the next accepted event's occurrences so the count stays true. Never construct an Issue or an ErrorEvent directly: services.record is the only writer and doing it by hand skips grouping, the regression flip, the event cap, the Sentry forward and the post-commit hooks.
- close_recovered — stapel_alerts.monitoring.close_recovered
  Close the issue of every expected check that did not fire this run, with fixed_in_version = 'recovered <ts>' rather than a release, because nobody deployed anything. Owner mode only — a reporter has no store to close anything in. A watchdog that can only OPEN issues produces a tracker full of blind spots that were fixed weeks ago, and a tracker nobody trusts is a tracker nobody reads.
- get_alerts_beat_schedule — stapel_alerts.beat.get_alerts_beat_schedule
  The beat entry for the watchdog, as a dict fragment to splat into CELERY_BEAT_SCHEDULE in the settings module itself — written there rather than merged from an on_after_finalize signal, so `manage.py check` (which reads settings.CELERY_BEAT_SCHEDULE) sees what beat will actually run. Pass seconds= to override the five-minute cadence; the interval is the resolution of every blind spot this finds.
- report — stapel_alerts.monitoring.report
  File each finding as a kind="monitoring" event through capture(), so a monitoring blind spot is grouped, counted and closed like any other bug in the tracker.
- reset_rate_limit — stapel_alerts.capture.reset_rate_limit
  Drop the per-fingerprint limiter's state. A TEST lever and the thing a reloaded worker wants: the limiter is process state, not database state, so a suite's rollback does not touch it and a carried-over window silently swallows the first capture of the next case.
- run — stapel_alerts.monitoring.run
  One pass of the monitoring blind-spot watchdog: collect, file, close what recovered, announce. Returns a summary and never raises, because a crashing watchdog is a blind spot of its own. Call it from your own scheduler if you are not using the shipped beat entry or the management command; do not reimplement the pass out of collect/report/close_recovered, since the ORDER is load-bearing (filing before closing is what records a check that flapped inside one run instead of dropping it).

## Extension points — what a product replaces, fork-free
- NOTIFY [import_string]
  The channel a deployment actually reads at 3am: a dotted path to notify(subject, body) -> bool, a callable, the shipped "telegram" name, or "none". It is used by two different mechanisms on purpose — a reporter digesting its buffer when the owner has been unreachable, and the monitoring watchdog announcing a blind spot — so a deployment configures one channel, not two.
- SerializerSeamMixin [subclass]
  Every view exposes request_serializer_class / response_serializer_class. Subclass the view, set the attribute, remount the URL — the way a host adds a field to an issue payload without forking the module.
- TARGETS / METRICS / HEARTBEAT_ALERT [merge_registry]
  What the watchdog looks at, inside STAPEL_ALERTS["MONITORING"]: scrape targets checked for up == 0 and for having no up series at all, metric names checked with absent(), and the alertname of an always-firing dead-man's switch. A bare target name matches on the job label; anything containing '=' is used as the label matcher verbatim, so a fleet keyed on instance needs no second setting.
- alerts.issue.opened [comm_event]
  The subscribe hook: a bug nobody had seen before now has a row, emitted once per Issue with the issue as the tracker knows it. A host pages, opens a ticket or posts to a channel off this instead of polling GET /issues. Deliberately NOT emitted per occurrence — that would put a failing loop's whole traffic on the bus and make every subscriber re-derive the grouping this module already did.
- alerts.issue.regressed [comm_event]
  The counterpart: an issue somebody marked fixed produced a new event. It carries the release that claimed the fix next to the release that is running, which is the pair that separates 'the fix is wrong' from 'the fix is not deployed'. Only the store ever sets this status — a caller able to assert a regression would be a caller able to decline to.
- alerts_exception_handler [settings_seam]
  REST_FRAMEWORK["EXCEPTION_HANDLER"] = "stapel_alerts.inputs.alerts_exception_handler". It delegates to core's stapel_exception_handler — the envelope stays core's and the response a client sees does not change — and captures what came back as a 5xx or as None. A host that would rather keep its own handler calls stapel_alerts.inputs.capture_exception(exc, context) from inside it.
- get_alerts_beat_schedule [callback]
  The blind-spot watchdog's schedule, shipped by the library as a splat a host merges into CELERY_BEAT_SCHEDULE (default: every five minutes). Shipped rather than documented because an unscheduled watchdog and a healthy fleet produce exactly the same output. Celery is optional: manage.py alerts_watch_monitoring is the same pass for cron.

## Fits with — fleet dependencies
- celery (optional) — the task_failure input, and the beat entry for the monitoring watchdog. Without it the input is a no-op and the watchdog is a management command any scheduler can run
- sentry-sdk (optional) — the export extra (pip install 'stapel-alerts[sentry]'): each event is forwarded and its id stored back. Storing locally is never conditional on it
- stapel-core (required) — AppSettings config layer, the fleet error envelope + error registry, IsStaffUser, the observability metrics facade and REDACT_FIELDS, trace ids, the GDPR provider registry, bus_event_parked (the structured DLQ input), comm deliver_to_subscribers (handler failures) and the comm outbox (alerts.issue.opened / .regressed)
- stapel-notifications (optional) — email/chat delivery of the three notification thresholds, and the telegram channel the fallback prefers when a deployment already has a bot wired in. Absent, the module sends the same message through its own direct Bot API call

## HTTP operations (5) — call by operationId, never by a typed path
Paths are relative to `/alerts/api/v1/`.
### alerts
- POST /issues/{issue_id}/fix — alerts_api_v1_issues_fix_create
- GET /issues — alerts_api_v1_issues_list
- PATCH /issues/{issue_id} — alerts_api_v1_issues_partial_update
- GET /issues/{issue_id} — alerts_api_v1_issues_retrieve
- POST /report — alerts_api_v1_report_create

## Error codes (49) — the StapelError envelope
Render `t(code, params)`; branch UX on the remediation. Localized text lives in docs/errors.<lang>.md, not here.
- error.400.alerts_batch_too_large [400] fix_input {max}
- error.400.alerts_invalid_report [400] fix_input
- error.400.alerts_status_not_settable [400] fix_input {status}
- error.400.bad_request [400] fix_input
- error.400.captcha_invalid [400] retry
- error.400.captcha_required [400] retry
- error.400.expected_list [400] fix_input
- error.400.field.blank [400] fix_input {field}
- error.400.field.does_not_exist [400] fix_input {field}
- error.400.field.invalid [400] fix_input {field}
- error.400.field.invalid_choice [400] fix_input {field}
- error.400.field.max_length [400] fix_input {field,max_length}
- error.400.field.max_value [400] fix_input {field,max_value}
- error.400.field.min_length [400] fix_input {field,min_length}
- error.400.field.min_value [400] fix_input {field,min_value}
- error.400.field.null [400] fix_input {field}
- error.400.field.required [400] fix_input {field}
- error.400.field.unique [400] fix_input {field}
- error.400.invalid_ad_id [400] fix_input
- error.400.validation_error [400] fix_input
- error.400.verification_failed [400] verify
- error.400.verification_invalid_factor [400] verify
- error.401.alerts_service_key_required [401] reauthenticate
- error.401.unauthorized [401] reauthenticate
- error.402.payment_required [402] retry
- error.403.alerts_service_key_invalid [403] retry
- error.403.forbidden [403] retry
- error.403.network_blocked [403] contact_support
- error.403.verification_enrollment_required [403] verify
- error.403.verification_required [403] verify
- error.404.ad_not_found [404] retry
- error.404.alerts_issue_not_found [404] retry
- error.404.not_found [404] retry
- error.404.verification_challenge_not_found [404] verify
- error.405.method_not_allowed [405] retry
- error.406.not_acceptable [406] retry
- error.408.request_timeout [408] retry
- error.409.conflict [409] fix_input
- error.410.gone [410] retry
- error.413.payload_too_large [413] retry
- error.415.unsupported_media_type [415] retry
- error.422.alerts_report_not_storable [422] wait_and_retry {events}
- error.422.unprocessable_entity [422] wait_and_retry
- error.423.locked [423] wait_and_retry
- error.423.verification_locked [423] wait_and_retry
- error.429.rate_limit [429] wait_and_retry {retry_after_minutes}
- error.429.too_many_requests [429] wait_and_retry
- error.500.internal [500] contact_support
- error.503.mandate_unavailable [503] retry
