Metadata-Version: 2.4
Name: keble-idea
Version: 0.10.3
Summary: Provider-neutral idea extraction, concept graph, generation, and search contracts.
Author-email: zma <bob0103779@gmail.com>
Requires-Python: >=3.13
Requires-Dist: keble-helpers<2,>=1.53.0
Requires-Dist: pydantic<3.0.0,>=2.10.0
Description-Content-Type: text/markdown

# Keble Idea

`keble-idea` owns the provider-neutral contracts and application behavior for
public content snapshots, standout ideas, semantic concepts, relationships,
immutable generations, durable work ledgers, and concept-first hybrid search.

It deliberately does not import FastAPI, Celery, MongoDB, Redis, Qdrant,
Neo4j, raw providers, or `keble-data-infra`. The host application implements
the protocols under `keble_idea.application.ports` and converts raw source
responses into idea-owned schemas at its outer boundary.

The standalone 0.10.0 intelligence-domain contract release expands that ownership from a
concept-only search package into the typed meaning layer for Keble's
intelligence data plane. It keeps immutable Content, standout Ideas, Concepts,
evidence, and lineage while adding scope-local Entities, reversible resolution,
product hierarchy, unified relationships, observations, Events, TrendEpisodes,
canonical commits, resource/cost dimensions, reprocessing, and a bounded query
DSL. It releases only provider-neutral contracts; Data Platform continues to own
adapter migration and coordinated integration gates.

The complete zero-context implementation and migration contract is
[`docs/INTELLIGENCE_DOMAIN.md`](docs/INTELLIGENCE_DOMAIN.md). The machine-readable
legacy mapping is
[`docs/compatibility/intelligence_domain_v1.json`](docs/compatibility/intelligence_domain_v1.json).

Version 0.10.2 adds `InitialRelationshipLineageCandidateV1`, a provider-neutral
initial-activation projection candidate. After a producer calls
`KnowledgeRelationshipVersion.validate_activation(...)` with loaded authority
facts, it may construct this candidate in the same preparation boundary. The
candidate requires one exact same-scope, same-commit, same-sequence, and
same-generation assertion/ACTIVE-version pair; it then emits only canonical
subject-to-object `RELATIONSHIP_ASSERTION`, assertion-to-Evidence
`EVIDENCED_BY`, and assertion-to-version `ACTIVATED_AS` edges. Assertion edges
retain original endpoints, predicate, and concrete node kinds, never resolved
endpoints. This is not a fabricated activation receipt or a re-resolution/
supersession contract: cross-commit evolution is rejected until a separate
policy defines it. The released lineage registry and its compatibility digest
remain unchanged.

Version 0.10.3 adds the finite `EXECUTE_SOURCE_PAGE` operation and its sole
`SOURCE_PAGE_EXECUTION` queue job type. Campaign `SOURCE_PAGE` work now admits
only that operation through the existing external-I/O gate; it no longer borrows
`PUBLISH_INTELLIGENCE` semantics. The contract remains provider-neutral: Data
Infra owns the authenticated execution boundary and Platform owns durable
attempt composition.

Version 0.10.1 adds the provider-neutral `LineageProjectionPolicyV1` candidate.
It is a finite, fail-closed publication policy rather than a graph-storage
schema: exactly ten public node families project from 11 admitted canonical
record kinds, while the other 14 record kinds are rejected before publication.
It names all eight public relations, deliberately leaves `DERIVED_FROM`,
`BOUND_TO_SOURCE`, and `GENERATED_BY` unsupported, and forbids `EventKind` as
generic edge metadata. Event kinds remain typed metadata on an `EVENT` node
with their existing ordered Entity-role contract. `RELATIONSHIP_ASSERTION`
requires an exact current `relationship-policy-v2` predicate and concrete
Entity/Concept node pair; the lineage registry pins that immutable V2 SHA-256
without changing either existing relationship-policy registry.

The version notes below describe the released 0.5.0–0.10.3 lines. Their
physical projection wording is historical; the store-neutral v1 intelligence
domain is released here, while its coordinated Platform release replaces the
multi-store intelligence path directly.

Version 0.5.0 changes immutable model execution from one physical deployment
to one Infisical-catalog logical model. Every model-backed envelope, receipt,
and accepted relationship freezes its purpose, `ProcessingTier`, selection
rule, logical model/provider keys, model revision, catalog fingerprint, and
generation. Every real provider request separately records a secret-free
`ProviderAttemptBinding` with endpoint-pool revision, endpoint, deployment,
region, and billing mode.

Version 0.6.0 aligns durable failure evidence with the universal raw contract:
a physical HTTP rejection may truthfully have complete zero billable usage
(for example reviewed authentication, permission, payment, invalid-request, or
rate-limit rejection). A zero-attempt local gate still cannot carry usage, and
`USAGE_ONLY` evidence remains financially incomplete.

Version 0.6.1 makes physical-attempt budgeting explicit on the immutable
failure snapshot. A zero-attempt `PLATFORM_RUNTIME` observation—such as waiting
for an already-paid completion projection—does not consume another provider
attempt. Provider/adapter failures and missing evidence remain conservative and
consume the budget. The wall-clock deadline still terminates local polling.

Version 0.6.2 completes the REVIEW2 replay boundary. Publication repositories
return one canonical hydrated result for both first publication and receipt
replay, so a redelivered paid extraction retains every idea, concept, and
posting identity needed for downstream projections. Durable upstream failure
evidence also keeps the authoritative classification origin and provider
mapping revision, while separately metered identity correction remains owned by
`ExtractionService` and is included exactly once in terminal receipt usage.

Version 0.7.0 introduces one evidence-backed semantic corpus for seller and
shopping discovery. COMMON and UNCOMMON ideas and facets pass the same quality
gates, persist normal concepts and postings, and share one vector/graph
projection. `SearchCorpusMode.DISTINCTIVE_ONLY` is the seller-safe default;
`ALL_DETERMINED` explicitly admits common purchase attributes for shopping
search. UNDETERMINED content retains compact assessment, receipt, usage, and
lineage facts but produces no searchable semantics. Content summary is derived
as COMMON, UNCOMMON, MIXED, or UNDETERMINED only after publication gates.

Version 0.8.0 combines the separate provider-neutral content catalog with the
durable restart-recovery contracts in one package candidate. Catalog projection
uses `PROJECT_CONTENT_SEARCH` plus `CONTENT_EMBEDDING`; recovery extends the same
canonical envelope with lease-renewal, expired-attempt planning, and finite sweep
reports. It adds no parallel job state machine and remains unreleased until the
coordinated Platform candidate and owner relevance gate are complete.

Version 0.9.0 releases the provider-neutral query-routing model-purpose
contract. `ModelPurpose.QUERY_ROUTING` freezes and serializes the logical model
selected for a bounded natural-language request without creating an
`IdeaJobType` or widening the closed queue lifecycle mapping. The final Idea
source authority for this contract is commit `fb90d06`; Platform retains route
selection, request orchestration, and durable query/session attachment.

The content-catalog contract adds a separate provider-neutral catalog-search
surface without changing semantic idea discovery. Both extraction
result variants now return up to 48 evidence-grounded human term units inside
the existing metered model call. `ContentSearchProfileBuilder` merges those
proposals with typed structured content facts by canonical NFKC/casefold
identity, preserves multi-word terms, caps only after merge priority, and builds
one deterministic token-measured embedding input for every classification,
including UNDETERMINED.

Content-profile publication is receipt-last: a terminal result is complete only
when its deterministic profile row also exists, and replay returns that exact
profile key. Host adapters implement the Idea-owned projection, ordered Mongo
hydration, Redis session, cursor-codec, and runtime-resolution protocols;
`ContentSearchService` composes them into the one new-search/cursor workflow.
It pre-generates the search-session identity used for semantic cost lineage,
probes one row beyond the configured result window, freezes only unique compact
frontier rows for fifteen minutes, and fills pages from ordered active Mongo
pairs without rediscovery. Exact admission always uses normalized keyword units;
semantic similarity can rank only that admitted set and `EXACT_ONLY` never
invokes query embedding. A generation-owned `ContentSearchRankProfile` changes
match/semantic/freshness weights independently from the pinned
`CONTENT_EMBEDDING` model, vector dimension, and Qdrant resource.
Content-search point UUIDs reserve generation bits while retaining canonical
content-key order inside a generation, so Platform can refine a newest-timestamp
boundary with Qdrant's stable point-ID order instead of accepting an arbitrary
subset of tied rows.

`VectorGenerationRecord` is the provider-neutral, immutable result passed to
Platform's `search_vectors_v1` conversion. It binds the resolved Content and
`CONTENT_SEARCH_PROFILE` refs, profile-input SHA-256, `CONTENT_EMBEDDING`
snapshot/resource, fixed v1 1,536-dimensional finite values, canonical vector
SHA-256, COSINE distance, source core frontier, vector descriptor, and
activation group. Its checksum preimage is exactly 1,536 IEEE-754 binary32
values, packed big-endian/network order as four bytes per dimension and
concatenated in order. The builder normalizes values through that encoding, so
a ClickHouse `Float32` readback reconstructs the same digest. A tombstoned
result is never semantically eligible. The stable key binds semantic
subject/profile/input/model-resource/vector-generation/descriptor authority,
not mutable frontier or activation lineage. Platform repositories own the
transactional unique `(record_key, record_version)` policy: an identical digest
rerun returns its existing row/version; a changed digest allocates exactly the
next contiguous version; a conflict fails and never overwrites history.

`INGEST` and `ENRICH` envelopes carry the tier before model selection;
model-backed envelopes carry it only inside `ModelExecutionSnapshot`. Platform
must resolve the logical snapshot before admission, select a physical endpoint
per attempt, and never substitute another logical model or tier during retry.
The removed physical snapshot fields and prior `model_id`, `error_code`, and
`retry_classification` contracts have no compatibility aliases.

`ModelPurpose.QUERY_ROUTING` identifies the logical model selected to translate
a bounded natural request into a typed query plan. It can be frozen and
serialized through `ModelExecutionSnapshot`, but it is not an `IdeaJobType` and
does not expand the queue envelope's closed job-to-purpose lifecycle mapping.
The host Platform owns request orchestration, route selection, and any durable
query/session attachment.

## Architecture

```text
provider DTOs / immutable source artifacts
        -> Platform conversion and visibility authorization
        -> keble-idea scoped records + canonical IntelligenceCommit
        -> Platform canonical writer and atomic activation
        -> snapshot-pinned query repository
```

`keble-idea` defines meaning, identity, invariants, checksums, and protocols. It
does not choose a database, table, index, queue name, provider client, router
model, or execution order. Data Platform owns those adapters and composes them
around these contracts.

Derived identities and lifecycle transitions are owner-type conversions:
`IntelligenceRecordIdentity.with_record_key` and
`RelationshipCandidateResult.as_shadow` reconstruct and validate the complete
schema. Production domain/application code must not use
`model_copy(update=...)`, because Pydantic does not validate updated values on
that path.

The v1 domain is organized around eight boundaries:

1. `IntelligenceScope` separates reusable PUBLIC truth from tenant-private
   truth. Every Entity, Concept, relationship, observation, evidence ref, merge,
   and query target carries that boundary. Repository maps use
   `ScopedRecordLookupKey`, which includes scope kind, scope key, record kind,
   and record key; a PUBLIC and tenant record cannot alias through a bare key.
2. `EntityVersion` distinguishes stable Brand, Storefront, Seller, Person,
   SocialAccount, ContentItem, ProductListing, ProductVariant, and ProductFamily
   identity. Raw/normalized terms, source bindings, policy/model revisions, and
   evidence remain append-only.
3. `EntityResolutionDecision` records create/bind/merge/reject/possible/reverse
   outcomes. Prefix overlap is candidate evidence only; accepted redirects do
   not delete the source record.
4. `KnowledgeRelationshipAssertion` selects an immutable versioned
   endpoint/evidence matrix. Historical `relationship-policy-v1` stays
   byte-stable; current `relationship-policy-v2` adds quality-gated
   Concept-to-Concept `RELATED_TO`, `BROADER_THAN`, `ENABLES`, and
   `APPLIES_TO`. `SAME_AS` remains absent: identity belongs to resolution,
   temporal changes belong to Events, and a completed zero-edge stage is a
   `RelationshipEvaluation`.
5. `IntelligenceCommit` groups bounded scoped mutations under canonical JSON,
   digests, counts, and immutable time. Physical batch serialization and
   activation mechanics remain adapter-owned.
6. `KnowledgeObservation`, Event, and TrendEpisode keep scalar facts, temporal
   occurrences, and noncausal precedence distinct.
7. `IntelligenceSchemaRegistry`, typed feature rows, and reprocess/rollback
   contracts make new metrics rebuildable without mutating old truth.
8. `IntelligenceQueryPlan` is a finite dumb-router output. Commutative groups
   normalize arbitrary model order; sort priority remains ordered. Standalone
   standout Ideas sort through the typed `idea.confidence` field, while
   TrendEpisodes sort through `trend.second_interval_end`, the end of their
   second half-open observation interval. Raw Content filters and sorts through
   `content.observed_at`, a DATETIME SearchProfile derived from its immutable
   observation instant.

Source campaign work is shard-and-page exact.
`SourcePageWorkIdentity` requires both `shard_ordinal` and `page_ordinal`
alongside its sole `work_key` and immutable request artifact. Neither ordinal
has a compatibility default. A continuation may advance only from a parent
identity in the same shard and at the immediately previous page; the outer
Platform campaign controller owns that comparison before queue admission.

Automatic Entity merge authority is intentionally narrower than candidate
generation. One endpoint-free `EntityResolutionQualityStratum` can be reused
only for later Amazon-to-Shopify fashion ProductFamily pairs whose canonically
loaded bindings, active Brand/category versions, market, model, prompt, cleaner,
and policy facts exactly match its persisted 500-pair/0.99-lower-bound gate.
The gate counts unique directional source/target pairs; multiple Evidence refs
for one pair cannot inflate that threshold. Every pair binds the exact stratum,
adjudication policy, and human outcome, and the gate recomputes its 95% Wilson
precision lower bound from those outcomes. SKU/model-prefix overlap remains
uncertain evidence and cannot create or merge an Entity by itself.

Canonical terms and source bindings are owned facts, not reusable name bags.
Every `CanonicalTermAssertion` names its exact Entity. A source binding is
unique by `(scope kind, scope key, channel, provider namespace, resource kind,
provider object id)`; ingestion service, evidence class, and market are
provenance. Provider adapters must encode provider-defined market identity into
the object id (for example `US:<ASIN>`). ProductListing activation additionally
fences the exact binding version, channel, provider, resource kind, market, and
currency, while ProductVariant remains canonical and has no direct source
binding.

Content-to-Entity authority is two-stage and version-exact. An accepted
`ContentEntityResolutionDecision` binds one immutable Content version to one
active Entity version; `ContentEntityLinkVersion` materializes that decision
with a semantic role. Relationship span/region evidence names those links, and
activation resolves each link, decision, Entity version, Content version, and
predicate-specific role. Bare Content evidence cannot authorize an Entity
endpoint.

Model relationship activation is corpus-backed. A
`RelationshipModelQualityGate` embeds exact distinct typed endpoint pairs,
human outcomes, Evidence membership, stratum checksum, and adjudication policy;
it recomputes its Lineage manifest, pair count, and two-sided 95% Wilson
precision bound. Repeated evidence for one pair, a swapped symmetric pair, or
an invented count/bound cannot unlock activation. Market/category strata are
validated from active source bindings and exact category-membership authority,
never caller booleans.

Coverage and time are reconstructible facts. `CoverageCountProof` hashes the
exact subject/group/window/cohort count basis; a multi-cohort aggregate requires
its own Lineage proof and cannot use one well-covered cohort to hide an
under-covered member. Events resolve exact Observation rows, metric registry and
derivation revisions, active Entity versions, every ordered subject role, source
period end, and the complete Evidence union. Frozen sequence definitions state
whether each leg needs any allowed rule or every rule per subject.

An ACTIVE reprocess run owns one complete multi-scope activation receipt. Each
scope proof binds its base/final frontier and activation group to the target
generation, and hot-search/vector rebuilds also bind the exact serving
descriptor artifact. Receipts and rollback previews persist canonical
self-digest-free body bytes plus their artifact checksum; rollback targets the
full receipt rather than reconstructing state from a partial pointer.
Reprocess page queue bindings carry a required immutable execution kind and
the exact page-plan or derived-action artifact digest. `INTERNAL_COMPUTE` and
`EXTERNAL_IO` authorize only their matching admission pair, while later
`DURABLE_RESULT_COMMIT` and `CONTROL_FINALIZATION` actions retain their bounded
no-I/O drain semantics. Provider/model pages therefore use the same owner gate
and resumable envelope as internal pages without granting external-I/O
authority from caller-supplied labels or to any other origin family.

Terminal visibility is proved from canonical storage, not from receipt-carried
hashes. Platform resolves the exact generation-pinned commits, activation
groups, assessment, and search profile into
`ContentPublicationActivationProof`; Idea then verifies every-and-only mutation
output and commit-set digest. Entity MERGE/REVERSE uses one
`EntityResolutionActivationReceipt`, so its decision, redirect, hierarchy
closure, and relationship revisions become visible together. Every variable
persisted collection is bounded and guarded by an introspective contract test.

Content Catalog and contextual Idea discovery remain separate application
profiles. The former admits evidence-grounded exact terms and may request typed
semantic fallback; the latter is Concept-first and retains distinctiveness,
postings, relationship expansion, and ranking explanation. The generic query
DSL adds Entity/Content/Idea/Concept targets, typed scalar/text/temporal/path
clauses, three-hop limits, finite registries, snapshot frontiers, evidence, and
explicit unavailable/insufficient-coverage outcomes. Router numeric inputs are
compiler-safe signed `Int64` or `Decimal(38, 12)` values; named discriminated
aliases preserve complete `valueKind`/`clauseKind` mappings in the generated
validation schema even when scalar clauses are reused inside relationship paths.
Temporal metrics retain this compact decimal shape and add one bounded
`currency` field: it is required exactly when `unit=CURRENCY` and forbidden for
every other unit, so CURRENT and DELTA price filters cannot compare implicit
currencies.

Hydrated query results expose logical values rather than ClickHouse row
projections. `matchedScalars` reconstructs the registered `ScalarQueryValue`
union, so text, money, dates, booleans, numbers, and scoped refs retain their
typed meaning. `matchedTemporals` carries the registered metric/statistic,
`DERIVED_FEATURE`/`OBSERVATION`/`EVENT` source family, value, unit and optional
currency, exact absolute interval, source channel, cohort, canonical
denominator, group-scope digest, member count, `CoverageResult`, Evidence refs,
and complete proof refs. `relationshipEndpoints` binds the same summaries to an
Entity/Concept that terminates a returned path. Each detail family is canonical
and bounded to 16 summaries; evidence/proof collections are canonical,
single-owner, snapshot-authorized, and bounded to 64 refs. Existing result-item
readers remain compatible because these additive collections default to empty.
Existing top-level `evidence`, `paths`, and `temporalSequences` collections are
also duplicate-free and canonical; path-internal node/predicate order remains
semantic and is never sorted.

Coverage failure is not an empty successful query. An
`INSUFFICIENT_COVERAGE` page contains canonical `coverageFailures`, each locating
the failed temporal calculation at `ROOT` or `RELATIONSHIP_ENDPOINT` and naming
the exact affected `subjectRef`. Root subjects use the finite result-record
families; endpoint subjects must be Entity or Concept. The page cannot carry
items, a cursor, or an unavailable reason, and it has no second page-level
coverage array: `coverageFailures[*].metric.coverage` is the canonical path. A
failure metric must be insufficient, while every root or endpoint
`matchedTemporal` must be sufficient. In contrast, a `COMPLETE` page with no
items and no failures is the valid zero-match result, while `UNAVAILABLE` keeps
its finite infrastructure reason vocabulary.

Contextual Concept expansion is revision- and checksum-pinned by
`SearchProfile`. It admits only the four v2 context predicates, keeps direct
`TAGGED` postings and their COMMON/UNCOMMON distinctiveness unchanged, and
bounds depth, per-origin edge fanout, expanded concepts, path confidence, and
the final frontier independently. `RELATED_TO` is symmetric;
`BROADER_THAN`, `ENABLES`, and `APPLIES_TO` retain direction in storage while a
query may deliberately read reviewed reverse adjacency. Every accepted context
edge requires `ProfileEvidence` and the existing 500-distinct-pair, 95% Wilson
lower-bound model-quality gate. See
[`docs/CONTEXTUAL_CONCEPT_RELATIONSHIPS.md`](docs/CONTEXTUAL_CONCEPT_RELATIONSHIPS.md).

Every generic query has one deterministic total order. Repositories apply the
router-authored logical fields in tuple order, then optional semantic score,
then the automatic `QUERY_RESULT_TIE_BREAK_V1` suffix: scope kind, scope key,
record kind, and record key, all ascending. The router never emits that scoped
identity suffix. Cursor continuation must compare the same components in the
same order, and a non-null cursor requires at least one item on its COMPLETE
page so an empty-page continuation loop cannot form. For raw Content, Platform
publishes `content.observed_at` into the
ordinary DATETIME SearchProfile from `ContentSnapshotRecord.observed_at`; the
field is not an adapter timestamp and does not make Content a graph endpoint.

Query identity and qualifying proof are deliberately separate. A
`ScopeServingFrontier` owns one typed `IntelligenceScope`; an
`IntelligenceSnapshot` binds the viewer, those exact frontiers, the
authorization-policy revision, and all projection/vector generations into
`snapshot_sha256`. `QueryProofAuthorization` then groups exact canonical proof
refs by their true owner scope and binds them to that digest. An authorized
tenant query may therefore return a PUBLIC result qualified by tenant-owned
relationship assertions, observations, Events, derived features, evidence,
Lineage, a canonical content-search profile, or ContentEntityLink. The same
result cannot replay that proof for a PUBLIC viewer or another tenant. Search
vectors remain repairable projections: semantic proof uses the canonical
content-search profile ref plus the snapshot-pinned vector generation/
descriptor, never an invented vector-record identity. Every
`QueryPathSummary` also carries the exact `qualified_result_ref`; this keeps a
Content, Idea, or TrendEpisode result bound to its underlying Entity/Concept
path without pretending those persisted identities are equal. Immutable
Content never becomes a graph endpoint. A raw Content result carries one
`QueryContentGraphBinding` whose exact ContentEntityLink authorizes a stable
CONTENT_ITEM Entity anchor at the start of the path through the canonical
`CAPTURE_OF` role.

Digest-critical query value objects are immutable through the full object
tree: `IntelligenceScope`, `ScopeServingFrontier`, and `ScopedRecordRef` are
frozen as well as their enclosing snapshot and proof receipt. Callers rebuild
and revalidate a new value instead of mutating a nested scope, epoch, or key
after its digest was accepted.

Unavailable query pages use one finite durable vocabulary:
`INTEGRITY_UNAVAILABLE`, `QUERY_EXECUTION_UNAVAILABLE`, or `SNAPSHOT_STALE`.
Temporary compiler migration guards are internal failures and must disappear
when their contract lands; they never become public reason codes.

An all-owned schema audit rejects unconstrained public strings, including list
items, keys, generations, cursor/session identities, and narrative fields.

## Development

Python 3.13 and `uv` are required.

```bash
uv sync
uv run pytest -q -m "not live and not slow and not eval and not local_stack and not db_stack and not container"
npx --yes pyright .
uv build
```

The portable test lane uses only reusable in-memory adapters from
`keble_idea.testing`. Stateful integration adapters and their isolation/cleanup
belong to the host Platform. Build verification also installs the wheel into an
empty environment and checks every name in `keble_idea.__all__`.

## Contract rules

- Persist only HIGH and MEDIUM ideas. LOW candidate details are discarded.
- Persist evidence-backed COMMON and UNCOMMON semantics through the same
  publication path; only UNDETERMINED content omits semantic projections.
- Publish one deterministic content-search profile for every active content
  classification; do not infer missing profiles from historical receipts.
- Admit catalog results only through exact canonical term units. Semantic
  similarity is an optional post-admission ranking input, never an admission
  fallback, and exact-only search incurs no query-embedding attempt.
- Keep catalog rank generations independent from content-embedding generations;
  rank-weight-only changes rebuild frontiers without rotating vector resources.
- Classify ideas and facets independently, then derive the content summary from
  the surviving common/uncommon counts rather than trusting a model-owned label.
- Default search to `DISTINCTIVE_ONLY`; require `ALL_DETERMINED` explicitly for
  shopping discovery, and pin the mode into the cursor session filter hash.
- Keep one concept vector and graph node across both modes. A common-only
  concept remains projected while its determined availability is positive.
- Write exactly one minimal receipt for every content/generation terminal state.
- Hydrate the complete canonical publication result on receipt replay; never
  return an empty derived-key set for already-published work.
- Keep receipts provider-neutral: physical token/image/duration usage remains in
  `ProcessingUsage`, while monetary pricing and per-content allocation belong
  exclusively to the platform-owned financial ledger. Receipt schema version 4
  rejects the removed `cost_micros` and loose `model_id` fields.
- Keep batch-identity correction in `ExtractionService`: one corrective call,
  one ordinal owned by the host, and one merged receipt usage total.
- Preserve the eight released semantic Concept values and the five
  generation-new CATEGORY/PRODUCT_CONCEPT/STYLE/ATTRIBUTE/PROBLEM values;
  provider/source metadata is not a Concept.
- Use `standalone_concept_error` at model boundaries so invalid facet labels
  receive a typed correction before publication applies the same strict policy.
- Merge only HIGH-confidence PURPOSE concepts automatically and keep it reversible.
- Vectorize concepts only unless a separately approved retrieval eval changes the design.
- Use immutable generations and active pointers; never mutate active indexes in place.
- Persist named, collection-owned integer-version migration checkpoints so outer
  workers resume by stable record key and never migrate a corpus at startup.
- Preserve original ASR and OCR independently from English-normalized fields;
  unknown-language OCR belongs in `ocr_text_original`, never guessed into
  `ocr_text_en` from the transcript language.
- Cutover only verified generations through compare-and-swap pointers and preserve
  the immediately previous verified generation for exact rollback.
- Report minimum-age or live-session-pin deletion guards as
  `GENERATION_RETENTION_BLOCKED`, never as an optimistic revision conflict.
- Stamp relationship coverage with the exact embedding, neighborhood, and
  relationship generations plus bounded neighbor similarities; a valid
  null/no-relation result still completes coverage. Persisted current/previous
  neighbor sets let outer adapters apply entry/exit hysteresis without all-pairs
  comparison.
- Keep `relationship_priority_score` and `relationship_priority_at` as bounded,
  provider-neutral scheduling hints. The platform derives them idempotently from
  posting quality/time so Stage-B graph budget reaches high-value/recent concepts
  first; they never change semantic concept identity or Stage-A search readiness.
- Compact alias postings asynchronously in bounded resumable pages; merge and
  unmerge correctness must never depend on compaction finishing first. Derived
  rows retain `origin_concept_key`, so unmerge removes them without deleting the
  original posting identities.
- Search is bounded, concept-first, count-free, and cursor-session based.
- Cursor signatures bind expiry, frontier revision, and the complete generation/profile identity.
- Operator lifecycle is canonical and typed (schema version 2): content moves
  `ACTIVE -> ARCHIVED -> ACTIVE` and enters `PURGING` terminally; ideas move
  `ACTIVE -> RETIRED -> ACTIVE` and enter `PURGING` terminally. `archived_at` /
  `retired_at` must match the state, transitions go through
  `assert_lifecycle_transition`, and public reads exclude non-ACTIVE rows.
- Operator job commands use the canonical status sets: retry targets
  `TERMINAL_JOB_STATUSES`, cancel targets `CANCELLABLE_JOB_STATUSES`
  (never LEASED work), and purge work routes through `IdeaJobType.PURGE`.
- Processing tiers are exactly `ECONOMY`, `BALANCED`, and `QUALITY`. They
  express requested cost/quality position, not provider identity. A retry keeps
  the frozen tier; changing it requires explicit reprocessing.

## Durable provider recovery

Each queue envelope snapshots the complete active
`ProviderRecoveryProfileSnapshot`, its matching `attempt_limit`, and an absolute
deadline. Automatic retry retains that snapshot. A manual retry also preserves
it unless the operator explicitly requests the active profile. Profile records
are append-only and activation uses a revision-guarded pointer; changing the
pointer affects future admissions only.

The initial profile is fixed at eight attempts, a 72-hour deadline, 30-second
fallback doubled to a one-hour cap, a 24-hour maximum provider hint,
deterministic 0-10% jitter, five unavailable failures in 60 seconds, a five-
minute open circuit, one half-open probe, and blocked release batches of 100.
Those are defaults for the first persisted profile, not module constants that
silently mutate existing jobs.

`retry_at_for_failure` anchors delay and jitter to the failure's immutable
observation time. Re-evaluating the same envelope attempt therefore returns the
same not-before timestamp. A provider hint above the submitted 24-hour policy
becomes `BLOCKED`; attempt/deadline exhaustion or permanent failure becomes
`DEAD_LETTERED`; operator action becomes `BLOCKED`. Only a retryable failure
within both budgets becomes `RETRY_SCHEDULED`.

`JobFailureSnapshot.consumes_attempt_budget` is the sole classification for
whether a leased retry counts as external work. Queue decision and persistence
must use it together: a zero-attempt Platform observation is evaluated against
the prior physical-attempt count and its lease increment is refunded atomically.
This preserves earlier paid attempts while allowing local projection polling
until the immutable deadline.

Restart recovery uses the same envelope identity. `QueueLeaseRenewal` describes
an exact owner/revision/attempt lease extension without implying a domain
revision, while `ExpiredEnvelopeRecoveryPlan` classifies an expired lease as
deferred, attempt reuse, fresh-attempt advancement, exhaustion, or a concurrent
resolution. A full financial-attempt census supplies the stable attempt keys to
abandon; a missing attempt is the only automatic-recovery case that may reuse
the queue attempt number. `QueueSweepReport` is the shared bounded operator
result and separates recovered work from live-claim deferrals, exhaustion, and
concurrent resolutions.

`UpstreamFailureEvidence` retains bounded status/code/request identity, opaque
quota scope, exact capacity, physical-attempt count, usage, and a completeness
bit. Empty usage is known zero only when the producer explicitly marks it
complete; incomplete empty usage and every `USAGE_ONLY` event remain billing
unknown.
Data Infra evidence also retains a provider-neutral classification origin plus
the upstream adapter's mapping revision separately from the Platform mapping
revision. This lets operators distinguish provider, contract, and Data Infra
runtime classification without making Idea depend on a provider package.
Payment requires a reviewed billing-code/account-status source; HTTP 402,
negative capacity, quota, and rate limiting are never sufficient by themselves.

Side effects if changes:

- Platform Mongo queue/profile/incident adapters and central indexes implement
  these exact field paths and compare-and-swap semantics;
- Data Platform job and commerce-operation APIs expose the same failure/policy
  facts through generated clients and translated frontend labels;
- finance records failed physical usage exactly once while pre-provider gate
  rejection remains zero spend;
- queue decision and persistence must never duplicate or disagree with
  `consumes_attempt_budget`;
- Platform must apply each expired-envelope plan in one queue/financial Mongo
  transaction and must not clone the envelope or create a parallel status;
- `EnvelopeProcessResult` must continue to contain exactly one result reference
  or failure snapshot.

## Images are a critical asset

Content payloads treat visuals as first-class evidence:
`ProductContentPayload.image_artifacts` carries model-ready product photos,
`CommerceProfileContentPayload.image_artifacts` carries acquired creator or
seller portraits, `ShortVideoContentPayload.thumbnail_artifacts` carries safe
ranked-video thumbnails without transient playback URLs, and
`VideoContentPayload` preserves uploaded video bytes plus key frames.
`IdeaSearchMedia.from_payload` projects the representative image/video (with
`IdeaSearchSourceSummary` for type-specific display facts) onto every search
item so consuming platforms can render media-rich cards. Artifact references
stay checksum-addressed (`ArtifactRef`) — delivery URLs are always resolved by
the serving platform at read time.

## Commerce discovery content

Trending commerce ingestion stays provider-neutral after the platform boundary:

```python
profile = CommerceProfileContentPayload(
    profile_type=CommerceProfileType.CREATOR,
    display_name="Rocket Lab Notes",
    image_artifacts=[acquired_profile_image],
)

video = ShortVideoContentPayload(
    title="A compact model-rocket launch guide",
    thumbnail_artifacts=[acquired_thumbnail],
)
```

Products keep `ProductContentPayload`; creators and sellers share the typed
profile payload with a stable `profile_type`; channel-wide ranked videos use a
separate short-video payload. Provider names, provider response bags, volatile
playback URLs, and raw enum labels do not enter these durable contracts.

## Language and standout evidence

`ContentBody.original_language` remains the backward-compatible ISO code.
Optional `ContentLanguageProvenance` explains whether it was source-declared,
text/media detected, or unresolved; a missing provenance on `und` identifies a
legacy row. Provider-neutral `ContentStandoutEvidence` records why a source
surfaced content without claiming profitability. Performance observations are
identified by metric kind, unit, observed time, and window: equal duplicates
collapse to the standout-linked evidence, while conflicting values are rejected.
