TEMPORARY WORKING NOTES — CODEX 0.150.1 TO 0.151.0 JSONL MIGRATION AUDIT
===============================================================================

Status
------

The read-only source audit is complete for TwiCC's current semantic behavior.
No TwiCC implementation change is part of this work.

Scope
-----

Audit every TwiCC call site that reads, classifies, searches, links, or derives
data from Codex JSONL records. Compare the legacy 0.150.1 records with the
paginated 0.151.0 records and the exact legacy-to-paginated migration output.

Explicitly out of scope for this audit:

- Supporting forked rollout lineages.
- Supporting reverted rollout lineages.
- Implementing any code change.

Repository baselines
--------------------

- TwiCC: /home/twidi/dev/twicc-poc
- Codex: /home/twidi/dev/codex
- Legacy Codex tag: rust-v0.150.1 (verified)
- Paginated Codex tag: rust-v0.151.0 (verified)

Confirmed format facts
----------------------

- A normal paginated session has one JSONL file.
- That file remains append-only during normal session use.
- A migrated legacy session is rewritten once into canonical paginated form.
- After migration, normal writes append to the migrated JSONL.
- The first SessionMeta record declares the history mode.
- Paginated records have a stable rollout ordinal.
- Codex also projects selected canonical history into thread_history_1.sqlite.
- JSONL remains the durable source of truth.
- SQLite is a rebuildable projection for paginated reads and search.

Initial TwiCC audit surface
---------------------------

Primary ingestion and computation:

- src/twicc/providers/codex/compute.py
- src/twicc/providers/codex/initial_sync.py
- src/twicc/providers/codex/sessions_watcher.py
- src/twicc/providers/sessions_watcher.py
- src/twicc/providers/compute_base.py
- src/twicc/sync_helpers.py

Derived data and cross-line lookups:

- ToolResultLink construction and enrichment.
- AgentLink construction and enrichment.
- Goal/task reconstruction.
- Plan document detection.
- Token usage and cost calculation.
- Context-window and model/runtime metadata.
- Image generation and screenshot association.
- Search indexing and search result line references.
- API/session serialization that exposes line_num.
- Frontend consumers that use lineNum as item identity or ordering.

Known broad migration rule
--------------------------

Legacy persisted display events are replaced by canonical ItemCompleted records.
Raw persisted ResponseItem records remain subject to the common persistence
policy. The audit must not assume that an old event name still exists.

Important distinction
---------------------

The audit must separate these cases:

1. A legacy event disappears, but its semantic data moves into ItemCompleted.
2. A raw ResponseItem remains and TwiCC should continue reading it.
3. Both legacy records represented the same operation and one must now be ignored
   to avoid duplicate UI items or links.
4. A record remains unchanged across both formats.
5. A field is normalized or synthesized by migration.
6. A field is genuinely unavailable after migration.

Audit completion
----------------

The repository search found JSONL-derived behavior in more than compute.py.
The audit completed these checks:

1. Verified the exact Codex tags and compared their persistence policies.
2. Enumerated direct JSON parsers and SessionItem content readers in TwiCC.
3. Enumerated legacy Codex event subtypes used by those readers.
4. Mapped each disappearing subtype through Codex's migration code.
5. Recorded each direct and indirect TwiCC consumer.
6. Inspected Codex migration tests and ran the relevant TwiCC baseline tests.

Audit findings
--------------

1. Text search and title/message extraction break on paginated rollouts.

`src/twicc/providers/codex/helpers.py::extract_indexable_text()` only accepts:

- `event_msg.user_message`
- `event_msg.agent_message`

Paginated history suppresses these legacy display events. The same user and
assistant content is stored in `event_msg.item_completed.payload.item`, using
canonical `UserMessage` and `AgentMessage` turn items. The helper must learn
that representation, directly or through a shared normalization layer.

This affects at least:

- search indexing;
- extraction of user messages;
- extraction of indexable user/assistant messages;
- any title or summary logic built on those helpers.

2. Several core inputs remain present with the same broad role.

These records are not replaced by `item_completed`:

- `session_meta`;
- `turn_context`;
- `event_msg.token_count`;
- `event_msg.thread_goal_updated`;
- raw `response_item.function_call` and `response_item.custom_tool_call`;
- raw `response_item.function_call_output` and
  `response_item.custom_tool_call_output`;
- top-level `compacted` records.

Their serialized names and the fields used by TwiCC remain compatible between
the two tags. They are not part of the legacy-display-event replacement.

3. High-risk legacy display events used by TwiCC move to canonical turn items.

Current TwiCC logic explicitly reads these legacy events:

- `user_message`;
- `agent_message`;
- `patch_apply_end`;
- `mcp_tool_call_end`;
- `image_generation_end`;
- `sub_agent_activity`.

Paginated history suppresses those event lines. Their useful information is
stored under `event_msg.item_completed.payload.item`. Every consumer must be
mapped to the corresponding canonical item subtype.

Correction: TwiCC does not currently depend on legacy `agent_reasoning` for
its visible reasoning card. It reads raw `response_item.reasoning`, which the
paginated persistence policy keeps. Migrated `agent_reasoning` records do
become canonical `Reasoning` items, but adopting them would require duplicate
handling. It is not required to preserve TwiCC's current behavior.

4. Existing raw tool call and raw tool output support is still useful.

The paginated persistence policy still stores supported raw `ResponseItem`
records. TwiCC can keep its existing raw function/custom call and output
parsing. It must also handle canonical completed items for outcomes that were
previously read from legacy end events.

5. Physical `line_num` remains usable after a full re-ingest.

Many TwiCC routines query earlier rows using `line_num`. These queries remain
valid for a normal append-only paginated file after migration if TwiCC deletes
and rebuilds every derived row for that session. Their content filters can still
fail, because several filters look for legacy payload types.

The audit must therefore distinguish two independent concerns:

- physical line ordering and stable line numbers;
- payload-type recognition at those line numbers.

Required deliverable format
---------------------------

The audit is not complete when it only identifies code that stops matching.
For every legacy line shape consumed by TwiCC, it must identify:

1. the exact legacy outer type and payload subtype;
2. the exact paginated replacement line;
3. the exact field path carrying each required value;
4. whether migration copies, converts, combines, synthesizes, or drops it;
5. every TwiCC consumer that must switch to the replacement;
6. whether a raw `response_item` remains and would cause duplicate handling.

Central replacement matrix for TwiCC
------------------------------------

This is the core answer to the audit. It lists only legacy JSONL records whose
semantic content TwiCC currently uses and whose legacy physical record is no
longer the source in paginated history.

| Information TwiCC needs | Legacy record currently read | Paginated record to read instead | Canonical data root |
| --- | --- | --- | --- |
| Human text and attachments | `event_msg` + `payload.type == "user_message"` | `event_msg` + `payload.type == "item_completed"` + `payload.item.type == "UserMessage"` | `payload.item` |
| Assistant text and message metadata | `event_msg` + `payload.type == "agent_message"` | `event_msg` + `payload.type == "item_completed"` + `payload.item.type == "AgentMessage"` | `payload.item` |
| Applied patch outcome and changed files | `event_msg` + `payload.type == "patch_apply_end"` | `event_msg` + `payload.type == "item_completed"` + `payload.item.type == "FileChange"` | `payload.item` |
| Rich MCP result and error | `event_msg` + `payload.type == "mcp_tool_call_end"` | `event_msg` + `payload.type == "item_completed"` + `payload.item.type == "McpToolCall"` | `payload.item` |
| Spawn call to subagent thread/path bridge | `event_msg` + `payload.type == "sub_agent_activity"` | `event_msg` + `payload.type == "item_completed"` + `payload.item.type == "SubAgentActivity"` | `payload.item` |
| Generated image, prompt, and saved path | `event_msg` + `payload.type == "image_generation_end"` | `event_msg` + `payload.type == "item_completed"` + `payload.item.type == "Extension"` + `payload.item.kind == "image_gen.generation"` | `payload.item` |

These are replacement readers, not additional readers. For the six semantic
roles above, the paginated parser must stop waiting for the old subtype and use
the matching canonical item.

The selection algorithm is therefore exact:

1. Read an outer line whose `type == "event_msg"`.
2. Require outer `payload.type == "item_completed"`.
3. Inspect `payload.item.type`.
4. Accept only the six semantic cases above.
5. For image generation, also require the Extension `kind` discriminator.

It is incorrect to treat every `item_completed` line as a visible message or
tool result. The paginated file also contains canonical items that duplicate
raw model-facing records TwiCC already reads.

`event_msg.web_search_end` also becomes an `item_completed/WebSearch` item
during migration. TwiCC intentionally ignores the old event today because it
cannot pair it with the persisted raw `web_search_call`. Therefore TwiCC does
not need to read the canonical `WebSearch` item to preserve current behavior.

The raw `response_item` calls, outputs, reasoning, inter-agent messages, and
plan source records that TwiCC already uses remain separate records. They are
not replacements for the six rows in this matrix.

Why the six-row list is exhaustive
----------------------------------

Codex 0.151's persistence policy gives the complete set of legacy-only durable
events that paginated history suppresses:

- `UserMessage`;
- `AgentMessage`;
- `AgentReasoning`;
- `AgentReasoningRawContent`;
- `EnteredReviewMode`;
- `ExitedReviewMode`;
- `PatchApplyEnd`;
- `ContextCompacted`;
- `McpToolCallEnd`;
- `WebSearchEnd`;
- `ImageGenerationEnd`;
- non-completed `SubAgentActivity`.

The intersection with information TwiCC actively consumes is exactly:

- `UserMessage`;
- `AgentMessage`;
- `PatchApplyEnd`;
- `McpToolCallEnd`;
- `ImageGenerationEnd`;
- `SubAgentActivity(kind == started)`.

The excluded legacy-only events do not require a replacement reader for current
behavior:

- reasoning: TwiCC reads the persisted raw `response_item.reasoning` instead;
- review-mode events: TwiCC does not classify or derive data from them;
- context-compacted event: TwiCC reads the persisted top-level `compacted` line;
- web-search end: TwiCC deliberately treats the raw web-search call as
  resultless and ignores the end event;
- completed subagent activity: TwiCC uses the raw `FINAL_ANSWER` completion
  envelope, not the completed activity item.

This intersection is the proof that no seventh legacy event replacement is
needed to preserve TwiCC's current semantic behavior.

Canonical wrapper used by the migration
---------------------------------------

Converted legacy display events are written as:

`event_msg.item_completed`

The wrapper contains:

- `thread_id`;
- `turn_id`;
- `item` (a tagged canonical `TurnItem`);
- `started_at_ms`;
- `completed_at_ms`.

The item subtype and its fields replace the useful payload of the old event.
The completed mappings follow for every subtype and TwiCC consumer.

Verified legacy-to-paginated mappings
--------------------------------------

The canonical `TurnItem.type` values are case-sensitive PascalCase strings.
They are not snake_case.

### `event_msg.user_message`

Replacement:

`event_msg.item_completed` with `payload.item.type == "UserMessage"`.

Field mapping:

- old `payload.message` -> concatenate text from new
  `payload.item.content[*]` entries whose `type == "text"`, using their `text`;
- old `payload.text_elements` -> the matching new text entry's
  `text_elements`;
- old `payload.images[*]` -> new content entries with `type == "image"` and
  `image_url` plus optional `detail`;
- old `payload.local_images[*]` -> new content entries with
  `type == "local_image"` and `path` plus optional `detail`;
- old `payload.audio[*]` -> new content entries with `type == "audio"` and
  `audio_url`;
- old `payload.local_audio[*]` -> new content entries with
  `type == "local_audio"` and `path`;
- old `payload.client_id` -> new `payload.item.client_id`;
- no old item id -> migration synthesizes `payload.item.id` as `item-N`;
- turn association -> outer new `payload.turn_id`.

Migration writes an empty-text user item if attachments exist. A text-only
consumer must concatenate all text entries with no separator, not assume one
entry.

### `event_msg.agent_message`

Replacement:

`event_msg.item_completed` with `payload.item.type == "AgentMessage"`.

Field mapping:

- old `payload.message` -> concatenate new
  `payload.item.content[*].text` where content `type == "Text"`, with no
  separator;
- old `payload.phase` -> new `payload.item.phase`;
- old `payload.memory_citation` -> new `payload.item.memory_citation`;
- old `payload.delivery` -> new `payload.item.delivery`;
- no old item id -> migration synthesizes `payload.item.id` as `item-N`;
- turn association -> outer new `payload.turn_id`.

The migration drops an empty legacy agent message instead of creating an empty
completed item.

### `event_msg.patch_apply_end`

Replacement:

`event_msg.item_completed` with `payload.item.type == "FileChange"`.

Field mapping:

- old `payload.call_id` -> new `payload.item.id`;
- old `payload.changes` -> new `payload.item.changes`;
- old `payload.status` -> new `payload.item.status`;
- old non-empty `payload.stdout` -> new optional `payload.item.stdout`;
- old non-empty `payload.stderr` -> new optional `payload.item.stderr`;
- old `payload.turn_id` -> outer new `payload.turn_id`;
- old `payload.success` is not copied. Success must be derived from new
  `payload.item.status`;
- TwiCC's DB-only `payload.original_files` enrichment is not part of the
  original Codex JSONL and is not produced by Codex migration.

### `event_msg.mcp_tool_call_end`

Replacement:

`event_msg.item_completed` with `payload.item.type == "McpToolCall"`.

Field mapping:

- old `payload.call_id` -> new `payload.item.id`;
- old `payload.invocation.server` -> new `payload.item.server`;
- old `payload.invocation.tool` -> new `payload.item.tool`;
- old `payload.invocation.arguments` -> new `payload.item.arguments`, with
  JSON null when absent;
- old successful `payload.result.Ok` -> new optional `payload.item.result`;
- old failed `payload.result.Err` -> new `payload.item.error.message`;
- success/failure -> new `payload.item.status`, serialized as
  `completed` or `failed`;
- old `payload.duration` -> new `payload.item.duration`;
- connector, app, plugin, link, resource URI, action, and read-only metadata
  move to same-named item fields, with camelCase serialization where declared.

The MCP status needs one special rule. A legacy `result.Ok` can contain a
CallToolResult whose `isError == true`. Migration then retains that body under
`item.result` but sets `item.status == "failed"`. `item.error` can remain null.
TwiCC must inspect both `item.error.message` and `item.result.isError`.

The optional serialized metadata names include `connectorId`,
`mcpAppResourceUri`, `linkId`, `appName`, `actionName`, `pluginId`, and
`readOnlyHint`.

### `event_msg.sub_agent_activity`

Replacement:

`event_msg.item_completed` with `payload.item.type == "SubAgentActivity"`.

Field mapping:

- old `payload.event_id` -> new `payload.item.id`;
- old `payload.kind` -> new `payload.item.kind`;
- old `payload.agent_thread_id` -> new `payload.item.agent_thread_id`;
- old `payload.agent_path` -> new `payload.item.agent_path`;
- turn association -> outer new `payload.turn_id`.

For a `kind == "started"` record, the new item id remains the spawning
`spawn_agent` call id. TwiCC can therefore rebuild the same `AgentLink`.
The old `occurred_at_ms` field is not copied into the item. The wrapper's
`completed_at_ms` comes from the old line timestamp.

### `event_msg.image_generation_end`

Replacement:

`event_msg.item_completed` with both:

- `payload.item.type == "Extension"`;
- `payload.item.kind == "image_gen.generation"`.

Field mapping:

- old `payload.call_id` -> new `payload.item.id`;
- old `payload.status` -> new `payload.item.status`;
- old `payload.revised_prompt` -> new camelCase
  `payload.item.revisedPrompt`;
- old `payload.result` -> new `payload.item.result`;
- old `payload.transparent_background` -> new camelCase
  `payload.item.transparentBackground`;
- old `payload.failure` -> new `payload.item.failure`;
- old `payload.saved_path` -> new camelCase `payload.item.savedPath`.

### Legacy reasoning events

`event_msg.agent_reasoning` and `event_msg.agent_reasoning_raw_content` become
`item_completed` records with `payload.item.type == "Reasoning"`. The migration
accumulates their text under one synthesized item id and can write multiple
completed records containing progressively larger arrays:

- summary text -> `payload.item.summary_text`;
- raw text -> `payload.item.raw_content`.

TwiCC currently renders raw `response_item.reasoning` records instead. Those
raw records remain persisted. Reading both representations would duplicate
reasoning, so no replacement is currently required for behavior parity.

Affected TwiCC consumers by replacement item
---------------------------------------------

This is a call-site inventory. A central normalization layer could shield many
of these call sites, but each behavior must still be tested independently.

### Canonical `UserMessage`

Backend consumers:

- `codex.compute._event_msg_text` and
  `CodexSessionCompute.extract_user_message_text`: initial session title and
  subagent prompt fallback;
- `CodexSessionCompute.compute_item_kind`: `ItemKind.USER_MESSAGE`;
- `CodexSessionCompute.analyze_content`: visible and searchable text;
- `CodexSessionCompute._restore_plan_prefix`: `/plan <prompt>` restoration;
- `codex.compute._is_internal_resume_message`: hide TwiCC recovery traffic;
- `CodexSessionCompute.extract_goal_event`: recognize a transformed
  `/goal clear` command. This specific source is TwiCC-generated and can stay
  in TwiCC's private normalized shape;
- `codex.helpers.extract_indexable_text`: search, history, and title inputs.

Indirect backend consumers through those hooks:

- batch compute title extraction and `user_message_count`;
- live watcher title extraction and `user_message_count`;
- startup and live Tantivy indexing;
- full search reindex;
- `/api/.../user-messages/` message history;
- CLI `session messages`;
- title-suggestion prompt collection;
- subagent prompt matching in the generic compute base.

Frontend consumers:

- `codex/Message.vue`: text and image attachments;
- `providers/codex/helpers.js::extractUserMessageText`;
- `providers/codex/helpers.js::extractUserMessageAttachmentCount`;
- optimistic-message reconciliation;
- in-flight send audit and failed-send recovery;
- message history and retry/resend text extraction;
- `ApiError.vue` recovery of the previous user text.

Important attachment change:

- legacy user images were sibling arrays such as `payload.images` and
  `payload.local_images`;
- canonical user inputs are interleaved entries in `payload.item.content`;
- the frontend must preserve their order or deliberately keep the current
  “images before text” presentation;
- attachment-only messages must still count as real user messages and match
  optimistic sends by attachment count.

### Canonical `AgentMessage`

Backend consumers:

- `CodexSessionCompute.compute_item_kind`:
  `ItemKind.ASSISTANT_MESSAGE`;
- `CodexSessionCompute.analyze_content`: visible and searchable text;
- `codex.helpers.extract_indexable_text`: assistant search indexing;
- `_transform_inline_provider`: screenshot-tag substitution currently reads
  legacy `payload.message`;
- streamed-item UUID attachment in `codex.helpers.enrich_live_items_payload`
  currently recognizes the legacy assistant kind/content combination and must
  be checked against the canonical line after classification.

Frontend consumers:

- `codex/Message.vue`: assistant text;
- `codex/AssistantMessage.vue`: normal Markdown and proposed-plan panel;
- `PlanImplementationBody.vue::latestProposedPlan`: reads the latest assistant
  text directly from `payload.message`;
- full-text search result navigation continues to use the physical line number
  of this canonical completed item.

Plan special case:

- raw `response_item.message role=assistant` remains persisted;
- an existing `item_completed` `Plan` is also persisted in both history modes;
- TwiCC currently exposes a proposed plan by transforming the raw response
  message into a private legacy-shaped assistant message;
- blindly rendering canonical `Plan` as well would duplicate the plan;
- behavior parity can keep the existing raw-response transformation and ignore
  `TurnItem::Plan`, or replace it only with explicit de-duplication.

### Canonical `FileChange`

Backend consumers:

- `_event_msg_call_id` equivalent: use `payload.item.id` to pair the result;
- `is_tool_result_item`;
- `extract_tool_result_info`;
- `_patch_apply_error`: derive from `item.status` and `item.stderr`, because
  canonical items have no `success` field;
- `analyze_content`: build the structured `ToolResultLink`;
- batch `_remap_orphan_end_event` and live
  `_lookup_orphan_end_exec_call_id`: nested code-mode patch pairing;
- `compute_link_extra`: aggregate lines and file statistics from
  `item.changes`;
- `extract_paths_from_tool_uses`: Git repository resolution;
- `extract_doc_edit_events`: plan/spec/handoff document tracking;
- `transform_tool_result_with_cache`: inject live `original_files` data;
- `codex.helpers.get_tool_results`: return the linked rich result to APIs and
  shares.

Frontend consumers:

- `providers/codex/toolHelpers.js`: expected result count, end-result lookup,
  patch paths, status, and rich-result selection;
- `codex/ApplyPatchContent.vue`: direct linked-line lookup, changes,
  `original_files`, full-file diff, hunk fallback, and per-file statistics;
- owner and share tool-result endpoints expose the same linked row;
- share `_result_items` fetches the DEBUG_ONLY row for large diff data.

Migration-specific loss risk:

`original_files` does not exist in Codex's legacy JSONL. TwiCC injected it into
the database during the original live run from an in-memory pre-tool cache.
Codex migration cannot recreate it. If TwiCC deletes all existing
`SessionItem` rows before rebuilding a migrated session, historical full-file
patch context disappears. The canonical `changes` data remains, so the UI can
fall back to reconstructed hunks.

Accepted product decision: discard historical `original_files` during a
legacy-to-paginated rebuild. Do not add a preservation or call-id transfer
path. New live FileChange rows can continue receiving `original_files` from the
existing in-memory cache when available.

### Canonical `McpToolCall`

Backend consumers:

- completed item id replaces legacy `call_id` for `ToolResultLink` pairing;
- `is_tool_result_item` and `extract_tool_result_info`;
- MCP error extraction now reads `item.status`, `item.error.message`, and
  `item.result.isError` instead of Rust `Result` tags `Ok` / `Err`;
- batch and live nested code-mode remapping must build the qualified MCP name
  from `item.server` and `item.tool`, not legacy `payload.invocation`;
- `analyze_content` and `codex.helpers.get_tool_results`;
- screenshot lookup intentionally remains on raw function/custom call outputs,
  preventing duplicate images from the richer MCP item.

Frontend consumers:

- `providers/codex/toolHelpers.js::mcpEndDisplayResult` currently unwraps
  legacy `result.Ok` / `result.Err`;
- direct and code-mode MCP rich result selection currently searches for
  `type == "mcp_tool_call_end"`;
- expected-result count remains two when both the raw model-facing output and
  canonical completed MCP item are linked.

### Canonical `SubAgentActivity`

Backend consumers:

- `_parse_sub_agent_activity_started`;
- `is_tool_result_item` gate used to create a live `AgentLink`;
- `extract_agent_info_from_tool_result`;
- batch `analyze_content` side table from agent path to spawn call id;
- live `_lookup_spawn_call_id_for_agent_path` backward scan;
- batch and live rebinding of a later raw `FINAL_ANSWER` to the original
  `spawn_agent` call;
- subagent stop detection indirectly depends on that link chain.

The raw `spawn_agent` call and its raw output remain persisted. Only the v2
thread-id bridge moves from legacy `sub_agent_activity` to the canonical item.

### Canonical image-generation Extension item

Backend consumers:

- `compute_item_kind`: `ItemKind.IMAGE`;
- `analyze_content`: mark visible without making a `ToolResultLink`;
- the raw `response_item.image_generation_call` remains DEBUG_ONLY to avoid
  duplicate rendering.

Frontend consumer:

- `codex/ImageGeneration.vue` must read canonical camelCase fields under
  `payload.item`, rather than legacy snake_case fields directly under
  `payload`.

### Consumers that remain on unchanged raw or common records

No paginated replacement is needed for these current behaviors:

- usage and cost: `event_msg.token_count`;
- context window and subagent turn boundary:
  `event_msg.task_started` / `event_msg.task_complete`;
- cwd/model: `turn_context`;
- opening metadata: `session_meta`;
- compact divider: top-level `compacted`;
- Goal lifecycle: `event_msg.thread_goal_updated` plus raw Goal tool outputs;
- tasks: raw `response_item.function_call` or code-mode
  `custom_tool_call` for `update_plan`;
- shell/tool call cards and outputs: persisted raw `ResponseItem` call/output
  records;
- view-image screenshot lookup: raw function/custom outputs;
- subagent v1 completion and multi-agent v2 `FINAL_ANSWER`: raw response
  messages;
- reasoning display: raw `response_item.reasoning`.

Exact TwiCC source-reader change matrix
---------------------------------------

This section distinguishes direct JSON readers from indirect consumers. The
function names are the current TwiCC entry points.

### Shared canonical accessors required

A canonical-only source parser needs stable accessors for these concepts:

- completed item: match outer `event_msg/item_completed`, then return
  `payload.item`;
- user text: join every `UserMessage.content` entry of type `text` with no
  separator, matching Codex's own `UserMessageItem.message()`;
- assistant text: join every `AgentMessage.content` entry of type `Text` with
  no separator, matching Codex app-server's own history reducer;
- call id: `FileChange.id`, `McpToolCall.id`, and `SubAgentActivity.id`;
- canonical result item: return `payload.item` for `FileChange` and
  `McpToolCall`, rather than returning the outer item-completed payload;
- image-generation item: match `Extension` plus
  `kind == image_gen.generation` before reading its fields.

Central accessors avoid repeating the outer-wrapper and PascalCase checks in
compute, DB lookups, helpers, and frontend renderers. They do not require a
legacy JSONL parser.

### Direct backend readers that must change

User and assistant messages:

- `_event_msg_text`: replace the flat old-message reader with canonical
  UserMessage/AgentMessage text accessors;
- `_is_internal_resume_message`: test the canonical UserMessage text;
- `CodexSessionCompute.compute_item_kind`: classify canonical UserMessage and
  AgentMessage items;
- `CodexSessionCompute.extract_user_message_text`: use canonical user text;
- `CodexSessionCompute.analyze_content`: use canonical user/assistant text and
  preserve the same visibility rules;
- `CodexHelpers.extract_indexable_text`: index canonical user/assistant text;
- `CodexHelpers.get_user_messages` and `get_indexable_messages`: no new JSON
  branch is needed after `extract_indexable_text` changes;
- `_restore_plan_prefix`: prefix the first canonical user-text entry, or store
  a presentation override instead of mutating source content;
- `_lookup_prev_plan_context`: replace the old flat-message LIKE filter and
  parse check for TwiCC's private `/plan` marker;
- `_goal_context_payload` and `extract_goal_event`: update their compatibility
  path if TwiCC's private visible Goal boundary becomes a canonical user item;
- `_transform_inline_provider`: create canonical private UserMessage and
  AgentMessage data, or stop rewriting content and store provider-neutral
  presentation overrides;
- screenshot substitution inside `_transform_inline_provider`: read and update
  canonical AgentMessage text entries instead of `payload.message`.

File changes:

- `_event_msg_call_id`: replace the old end-event whitelist with an item-id
  accessor for canonical FileChange and McpToolCall;
- `_patch_apply_error`: match FileChange and derive success only from
  `item.status == "completed"`; read `item.stderr` for the detailed error;
- `_event_msg_payload_error`: dispatch canonical FileChange/McpToolCall items;
- `is_tool_result_item`: accept canonical FileChange as a structured result;
- `extract_tool_result_info`: use `item.id` and the canonical error helper;
- `remap_tool_result_id` and `_remap_orphan_end_event`: detect FileChange and
  read `item.id` plus `item.changes` for nested code-mode pairing;
- `remap_tool_result_id_live` and `_lookup_orphan_end_exec_call_id`: apply the
  same FileChange checks in the DB-backed live path;
- `extract_paths_from_tool_uses`: read keys from `item.changes`;
- `extract_doc_edit_events`: read canonical changes and status;
- `compute_link_extra`: compute file and line statistics from
  `item.changes`;
- `transform_tool_result_with_cache`: key the cache with `item.id` and attach
  TwiCC's `original_files` enrichment to the canonical row or separate storage;
- `analyze_content`: create the structured ToolResultLink from canonical
  FileChange;
- `CodexHelpers.get_tool_results`: return the canonical FileChange item for a
  linked result row.

MCP calls:

- `_mcp_end_qualified_name`: build `mcp__<server>__<tool>` directly from
  `item.server` and `item.tool`;
- `_mcp_tool_call_end_error`: use `item.error.message` and
  `item.result.isError`; do not rely only on `item.status`;
- `_event_msg_call_id`, `_event_msg_payload_error`, `is_tool_result_item`,
  `extract_tool_result_info`, and `analyze_content`: accept canonical
  McpToolCall and use `item.id`;
- `_remap_orphan_end_event`, `remap_tool_result_id_live`, and
  `_lookup_orphan_end_exec_call_id`: match canonical MCP server/tool fields
  for nested code-mode pairing;
- `CodexHelpers.get_tool_results`: return the canonical McpToolCall item.

Subagent activity:

- `_parse_sub_agent_activity_started`: match canonical SubAgentActivity and
  read `item.id`, `kind`, `agent_thread_id`, and `agent_path`;
- `is_tool_result_item`: keep allowing a started activity through the live
  AgentLink creation gate, although it remains `has_tool_result == false`;
- `extract_agent_info_from_tool_result`: return the same
  `(spawn call id, thread id, true)` tuple from the canonical item;
- `analyze_content`: rebuild the `agent_path -> spawn call id` side table and
  AgentLink data from the canonical item;
- `_lookup_spawn_call_id_for_agent_path`: change both its LIKE prefilter and
  its parse verification to canonical SubAgentActivity;
- later raw `FINAL_ANSWER` rebinding stays unchanged after that side table is
  populated correctly.

Image generation:

- `compute_item_kind`: classify only canonical Extension items whose kind is
  `image_gen.generation` as IMAGE;
- `analyze_content`: mark that item visible without creating a ToolResultLink;
- the raw `response_item.image_generation_call` remains SYSTEM/DEBUG_ONLY.

### Direct frontend readers that must change

User and assistant messages:

- `providers/codex/helpers.js::buildOptimisticUserMessageContent`: build the
  same canonical UserMessage shape as a real persisted item, or build a
  provider-neutral normalized message consumed by the same accessor;
- `extractUserMessageText`: join canonical `content[type=text].text` entries;
- `extractUserMessageAttachmentCount`: count canonical `image` and
  `local_image` entries;
- `codex/Message.vue`: read canonical user/assistant text and extract user
  image URLs from canonical content entries;
- `codex/PlanImplementationBody.vue::latestProposedPlan`: use the shared
  canonical assistant-text accessor instead of `payload.message`;
- `ApiError.vue`, optimistic reconciliation, in-flight-send recovery, and
  message retry logic already call the provider helper. They change
  indirectly when that helper changes.

File changes:

- `codex/ApplyPatchContent.vue::patchEndPayload`: find a linked
  `item_completed/FileChange` row, validate `item.id`, and return `item`;
- `providers/codex/toolHelpers.js::findCodexEndEventPayload` and
  `findPatchApplyEndPayload`: select canonical FileChange from the result
  array;
- path resolution, changes rendering, hunk fallback, and file statistics can
  keep their current field reads after they receive the canonical item root;
- expected result count stays two: one raw model-facing call output plus one
  canonical FileChange row.

MCP calls:

- `mcpEndDisplayResult`: select canonical McpToolCall, use `item.result`
  directly, and use `item.error.message` on failure;
- the direct-MCP and code-mode MCP result-presence checks must match
  `type == "McpToolCall"` instead of `mcp_tool_call_end`;
- expected result count stays two: one raw model-facing output plus one
  canonical McpToolCall row;
- `JsonHumanView` already understands the CallToolResult camelCase content
  shape and needs no schema-specific change.

Image generation:

- `codex/ImageGeneration.vue`: unwrap `payload.item`, require Extension plus
  the image-generation kind, and read `revisedPrompt`, `result`, `savedPath`,
  `transparentBackground`, and `failure` from that item.

### Indirect consumers that need rebuilt data, not a new JSON branch

These paths consume `SessionItem.kind`, `text_content`, link rows, or helper
output. They do not need to recognize `item_completed` themselves:

- initial and live user-message counts and titles;
- weekly/daily activity counters;
- search indexing, reindexing, snippets, and search navigation;
- `/user-messages/`, CLI `session messages`, and title suggestion;
- `tool_results_payload`, owner APIs, public shares, and share result fetches;
- generic visual grouping and conversation block boundaries;
- generic tool-state, result-count, and AgentLink APIs.

They still require an authoritative purge and rebuild after migration because
their current rows and line numbers were derived from the legacy file.

### Dormant legacy-name branches to remove, not port

`CodexHelpers._PERSISTED_END_EVENT_TYPES` still lists `web_search_end` and
`image_generation_end`, and frontend `PERSISTED_END_EVENT_TYPES` still lists
`image_generation_end`. Current compute never creates ToolResultLinks for
either event. These branches are unreachable in a clean current recompute.

The canonical parser should not add WebSearch or image-generation items to the
generic tool-result path merely to preserve these stale whitelists. Web search
stays resultless. Image generation stays a standalone IMAGE row.

JSONL rewrite hazard during Codex migration
-------------------------------------------

`read_session_items_from_file()` assumes every change is an append. It always
seeks to `Session.last_offset`. It does not detect that the file was replaced
or rewritten.

Consequences if Codex migrates a file while the normal TwiCC watcher handles
it:

- if the rewritten file is shorter, TwiCC seeks past EOF and imports nothing;
- if it is the same size or larger, TwiCC can read a suffix starting in the
  middle of the rewritten history and append it under incorrect line numbers;
- mtime and file-size checks do not prove prefix identity;
- `last_line` remains based on the legacy file;
- all links and computed rows can then point to unrelated physical lines.

Therefore migration must be a coordinated special path. For each legacy
session it must prevent normal incremental sync, run Codex migration, delete or
replace the session's source-derived rows, reset `last_offset` and `last_line`,
and ingest the complete migrated file from byte zero. A normal compute-version
bump alone is insufficient because it does not replace raw `SessionItem`
content or line numbers.

Why legacy and migrated line numbers cannot be mapped arithmetically
--------------------------------------------------------------------

The Codex canonicalizer does not preserve one output line per input line. It:

- moves the selected `session_meta` to the head and writes it with ordinal 0;
- skips later `session_meta` records;
- adds an `ordinal` to every output line;
- changes `history_mode` to `paginated`;
- clears `history_base` and `subagent_history_start_ordinal`;
- can synthesize `task_started` and `task_complete` around implicit turns;
- replaces legacy completion events with `item_completed` records;
- drops legacy display events that have no paginated persistence role;
- copies persisted raw `response_item` records;
- emits an extra completed `HookPrompt` after a matching raw user response;
- can emit several progressively accumulated `Reasoning` completed items for
  several legacy reasoning events;
- can assign a completed item to an explicit earlier turn without changing the
  selected event's semantic content;
- rejects the migration if a raw `ResponseItem::Other` is present.

Therefore no constant offset or old-to-new line-number formula exists. A
semantic key, such as a call id, can preserve selected relationships. Every
physical line-derived structure must otherwise be rebuilt.

Canonical paginated records that TwiCC should deliberately ignore for parity
-------------------------------------------------------------------------

The new history contains more canonical completed items than TwiCC currently
needs. Reading every `item_completed` line would create duplicates or new UI
behavior.

- `Reasoning`: ignore for parity. TwiCC already reads the preserved raw
  `response_item.reasoning` records.
- `Plan`: ignore for parity. TwiCC currently transforms the preserved raw
  assistant response carrying `<proposed_plan>` into its own visible plan.
- `FunctionCallOutput`: ignore for parity. TwiCC already consumes the preserved
  raw function/custom tool output.
- `CommandExecution`: ignore for parity. TwiCC builds shell cards and output
  links from preserved raw call and output records.
- `WebSearch`: ignore for parity. TwiCC currently exposes the preserved raw
  hosted web-search call without the result-rich legacy end event.
- `ContextCompaction`: ignore for parity. TwiCC uses the preserved top-level
  `compacted` record for the divider.
- `HookPrompt`, `EnteredReviewMode`, `ExitedReviewMode`, `DynamicToolCall`,
  `ImageView`, and other currently unsupported item types: keep them
  non-visible until TwiCC intentionally adds a feature for them.

This allow-list is important. The paginated parser must select only the
canonical replacement items listed earlier, plus the raw/common records that
TwiCC already consumes.

How to detect the source history mode
-------------------------------------

The head `session_meta.payload.history_mode` is the authoritative discriminator:

- missing field or legacy value: legacy history;
- `"paginated"`: paginated history.

Codex migration rewrites the selected metadata record with `"paginated"`.
TwiCC's current `initial_sync.extract_session_meta()` does not read this field.
The migration path needs to inspect it before normal incremental ingestion.

Line-number-derived TwiCC data that migration invalidates
---------------------------------------------------------

The following data stores physical JSONL line numbers. It must be deleted,
rebuilt, remapped, or explicitly invalidated after a rollout rewrite:

- `SessionItem.line_num`, with the unique `(session, line_num)` constraint;
- `Session.last_line` and `Session.last_offset`;
- `SessionItem.group_head` and `SessionItem.group_tail`;
- `ToolResultLink.tool_use_line_num` and `tool_result_line_num`;
- `AgentLink.tool_use_line_num`;
- `Session.tasks[*].line`, which records the source plan/task line;
- Tantivy documents and search hits keyed by `(session_id, line_num)`;
- frontend item arrays and the virtual-item cache keyed by `lineNum`;
- frontend expanded group state keyed by group-head line number;
- frontend detailed-conversation blocks keyed by user-message line number;
- frontend internal expanded groups keyed by outer item line number;
- frontend open detail panes and linked result-line caches;
- public snapshot shares using `Share.options.frozen_at_line`.

`ToolResultLink` and `AgentLink` do not have foreign keys to `SessionItem`.
Deleting a session's `SessionItem` rows does not cascade to these link rows.
The migration path must clear and rebuild them explicitly.

Frontend `unloadSession(session_id)` already clears most in-memory line-keyed
state. A migrated-session event must force that unload before the client fetches
the rebuilt metadata. Otherwise old expansion and link state can attach to
unrelated new lines.

Snapshot share problem
----------------------

A session snapshot stores the old physical boundary in
`Share.options.frozen_at_line`. All share item, tool-result, tool-state, and
root-subagent queries enforce that number.

After migration, the same number no longer denotes the same semantic point.
Leaving it unchanged can expose too much history or hide content that belonged
to the original snapshot. Codex migration does not provide an old-to-new line
mapping.

Possible policies require a product decision:

- block migration while snapshot shares exist;
- remap each snapshot boundary by replaying the old and new histories and
  matching a semantic boundary;
- materialize immutable snapshot content outside the live session;
- invalidate or re-freeze the share, which changes its meaning and therefore
  requires explicit user consent.

This is not solved by recomputing session metadata or links.

Accepted requirements:

- The rollout migration must be fully automatic.
- It cannot require owners to refresh or re-enable snapshot shares.
- It cannot add a table or a materialized snapshot store for this case.

Accepted timestamp-remapping policy:

1. Before deleting legacy SessionItems, inspect every snapshot share and its
   old `frozen_at_line`.
2. Find the last old item at or before that line with a valid timestamp. Store
   that exact timestamp as the migration anchor.
3. Persist the temporary anchor in the existing `Share.options` JSON. This
   makes crash recovery possible without a new table.
4. Migrate Codex and fully rebuild the session.
5. Set the new boundary to the greatest new line whose timestamp is `<=` the
   anchor timestamp.
6. Update `frozen_at_line` and remove the temporary anchor key atomically.

Codex preserves the source timestamp on copied and converted output lines.
Synthesized turn/item wrappers use the triggering source line's timestamp.
Output order remains chronological for normal sessions.

Local evidence: 1,665 Codex JSONL files containing 704,185 lines were checked.
No file had decreasing valid timestamps. Duplicate timestamps were common:
1,626 files had at least one duplicate group.

Accepted product decision: include every new line that shares the anchor
timestamp, even if the old `frozen_at_line` split that timestamp group. A small
amount of same-millisecond post-boundary content is acceptable during this
one-time migration. No stricter source-line mapping or fallback is required.

Simple migration envelope: canonical summary
--------------------------------------------

The JSONL parser is a black box in this section. The envelope sits in the
Codex main process between stale-session discovery and the metadata compute
queue.

Normal paginated session:

1. Detect that the source is already paginated.
2. Queue normal compute when its compute version is stale.
3. Let the watcher continue append-only ingestion from `last_offset`.

Legacy session:

1. Check TwiCC's live-agent state.
2. If active, add the session id to the in-memory deferred set and immediately
   continue with the next candidate. Do not hold a gate and do not queue it.
3. If inactive, enter the short per-session migration gate.
4. Save frozen-share timestamp anchors in existing `Share.options`.
5. Quiesce watcher ingestion for this session.
6. Invoke Codex migration. Codex's per-thread OS file lock is the final race
   guard.
7. On `skipped_busy`, change no line-derived TwiCC data and remove any temporary
   share anchors written for this attempt. Resume the watcher, release the gate,
   defer the session, and continue.
8. On success, discard all old line-derived TwiCC state, import the new JSONL
   from byte zero, run the full metadata rebuild, and rebuild links and search
   state.
9. Remap every frozen share to the greatest new line whose timestamp is less
   than or equal to its saved anchor. Remove the temporary anchor.
10. Commit the new offsets, line count, and compute version only after the full
    rebuild succeeds. Resume the watcher and release the gate.
11. From then on, treat the migrated file as an ordinary append-only paginated
    session.

Deferred scheduler:

- keep one Codex background-compute run alive while any Codex session still has
  a stale `compute_version`;
- the main-process compute coordinator skips active/busy sessions and continues
  immediately with other ready sessions;
- the spawned CPU worker stays alive and blocks idle on its existing command
  queue when no session is currently ready;
- a TwiCC agent reaching `DEAD` wakes the coordinator immediately;
- the same coordinator uses a timed wake-up to retry external Codex writers;
- stop the worker only when no stale Codex session remains;
- no separate retry scheduler and no durable deferred queue are needed. After
  restart, source format and `compute_version` rediscover unfinished work.

This long-lived run preserves the current single-writer architecture:

- the spawned CPU worker only reads TwiCC SQLite;
- it emits the normal result messages through the existing subprocess queue;
- the main-process DB writer applies all TwiCC SQLite writes;
- share-anchor writes, structural purge/reimport, and offset resets must also be
  represented as DB-writer jobs. The migration coordinator must not add a
  second direct SQLite writer;
- Codex's own migration subprocess writes Codex's JSONL and Codex SQLite, not
  TwiCC SQLite.

The main-process coordinator, not the CPU subprocess, owns readiness checks,
live-agent state, migration, watcher quiescence, and DB-writer jobs. Calling the
combination "the compute" is reasonable at the subsystem level, but the child
must remain a read-only calculator.

This matches the current command boundary exactly. Today
`start_background_compute_task` in the main process queries every stale session
id in descending `mtime` order. It sends only two command shapes to the child:

- `{"session_id": <id>}` to compute one session;
- `None` to stop the worker.

The separate multiprocessing stop event is only an out-of-band shutdown guard.
There are no pause, rescan, retry, migration, or progress commands today. The
worker does not discover its own candidate list.

Claude Code and Codex use this exact same provider-agnostic function. Each
provider creates its own `ComputeContext`, but both enqueue every stale session
id individually and then enqueue `None`. The long-lived scheduling behavior can
remain Codex-only through a provider policy/hook; Claude Code does not need to
change for this rollout migration.

This was an orchestration choice, not a SQLite limitation. The worker already
opens TwiCC SQLite read-only enough to load each named Session and its items.
Keeping candidate discovery in the main process lets the main task own the
fixed startup total, progress reporting, run id, provider lifecycle, ordering,
and final stop. The original pipeline was one-shot, so it could enqueue the
whole list once and then send `None` immediately.

Dispatch only one session at a time, or keep at most one command prefetched.
After the worker sends the direct `computed(session_id)` signal, query stale
sessions again in `-mtime` order and dispatch the next one without waiting for
the previous SQL apply. This lets a recently active session that just became
available run before an old backlog, while CPU work overlaps DB-writer work.
The separate `applied` signal is used for bookkeeping and the final stop
decision, not for releasing the worker to its next CPU job.

For maximum overlap, distinguish two per-session signals:

1. `computed(session_id)`: emitted by the worker immediately after it has put
   the normal result payload on the DB-writer result queue. Send this through a
   new direct per-run worker-to-coordinator status channel. The coordinator can
   choose and send the next session at this point, while the DB writer applies
   the previous session. Do not make the DB writer relay CPU-worker readiness:
   it is a separate task with a separate responsibility, and routing readiness
   through its serial apply loop would couple scheduling to SQL latency.
2. `applied(session_id, outcome)`: emitted by the DB writer after the SQL
   transaction succeeds, is skipped as stale, or fails. The coordinator needs
   this second signal before declaring the whole run complete.

There is no worker-to-coordinator channel today. The existing subprocess result
queue is worker-to-DB-writer only. Current run-level completion is indirect:

1. `arm_compute_completion` creates an in-memory asyncio Future in the main
   process and returns it to the coordinator.
2. The worker enqueues every `session_complete` or `error` to the DB writer.
3. After it consumes `None`, the worker enqueues `done` to that same FIFO.
4. The DB writer applies all earlier result messages, receives `done`, flushes
   run aggregates, and resolves the stored Future.
5. The coordinator wakes from `await done_future`.

The DB writer is not the coordinator. They are separate async tasks in the main
process. The indirect final Future is appropriate for drain completion, but it
is not a direct scheduling channel and there is no current per-session signal.

The coordinator must track submitted/computed-but-not-applied session ids. A DB
query still sees their old `compute_version`, so without that in-memory set it
would submit the same session twice while its result waits for SQL application.

No `Wait` command is required. An empty command queue already means wait: the
worker calls `get(timeout=0.5)` and loops while no command exists. If stale but
busy sessions remain, the coordinator sends nothing. It sends `None` only when
no candidate remains and every pending DB application has finished.

One current lifecycle dependency needs separation. Both provider orchestrators
set `compute_done` only when `start_background_compute_task` returns, and the
global search-indexing sweep waits for provider compute completion. A Codex
worker that stays idle for hours must not block that initial search sweep. The
Codex coordinator must signal `initial_compute_pass_done` after all immediately
processable startup sessions finish, while its long-lived deferred loop and
worker remain alive. Sessions rebuilt later must request their own search
reindex through the DB writer.

When a scan finds ready sessions, prepare and dispatch the newest ready one.
When every stale session is busy, wait on an `asyncio.Event` with a timeout:
agent `DEAD` sets the event, while timeout expiry provides the external-writer
poll. Do not spin. Provider shutdown uses the existing stop path and terminates
the idle worker.

Resume/send race:

- if resume wins first, Codex owns the writer lock and migration defers;
- if migration wins first, a TwiCC resume/send waits on that session's gate;
- unrelated sessions never share this gate.

Crash recovery:

- the in-memory gate and deferred set disappear safely;
- OS locks release when their owning process exits;
- Codex repairs interrupted publication with its `.pending` journal;
- TwiCC leaves `compute_version` stale until completion;
- a paginated source plus stale compute version means migration succeeded but
  the TwiCC rebuild must run again from byte zero;
- a retained share timestamp anchor makes frozen-share remapping resumable.

Concurrency between migration, compute, watcher, and live Codex writers
-----------------------------------------------------------------------

Codex 0.151 has two distinct cross-process file-lock layers:

- `$CODEX_HOME/.tmp/rollout-maintenance.lock` serializes migration and rollout
  compression for the whole Codex home;
- `$CODEX_HOME/thread-writer-locks/<thread-id>.lock` gives one process exclusive
  writer ownership of one thread.

The maintenance lock does not block live thread writers. Its source comment
explicitly says it is separate from the per-thread writer locks. The per-thread
lock is the data-integrity boundary relevant to TwiCC.

An app-server live recorder acquires the per-thread writer lock during thread
create/resume and retains its guard in the live-recorder entry until thread
shutdown. This includes an idle but still loaded TwiCC agent in USER_TURN.

Manual `migrate-rollouts --apply` behavior:

1. It takes the global maintenance lock.
2. It reads SessionMeta and identifies the thread.
3. It tries to take that thread's writer lock with a non-blocking file lock.
4. If a live recorder owns the lock, the outcome is `skipped_busy`. The original
   rollout remains byte-for-byte unchanged.
5. If migration owns the lock, it retains it through canonical staging, SQLite
   projection, verification, journal publication, atomic JSONL replacement, and
   cleanup.
6. A competing thread resume/create during that interval also uses a
   non-blocking acquire. It receives a conflict: `thread <id> already has an
   active writer`. It does not wait for migration automatically.

Therefore Codex prevents concurrent append/rewrite corruption whichever side
wins the race. It does not provide transparent user-message waiting or retry.
The CLI process can also exit successfully when an individual outcome is
`skipped_busy`; TwiCC must inspect the JSON outcome, not only the exit code.

Codex writes a durable `.pending` migration journal before publishing. A later
apply run repairs a published rollout whose SQLite projection was interrupted.
Recovery also returns `skipped_busy` while another writer owns the thread.

TwiCC activity detection
------------------------

The JSONL watcher is not an authoritative active-session detector. It knows
that new bytes arrived and touches activity on an already-known live agent. An
idle live recorder can hold Codex's writer lock without changing the JSONL.

For TwiCC-owned agents, the in-memory Codex agent manager and its non-DEAD
ProcessRun are the authoritative fast pre-check. For an external Codex process,
only Codex's cross-process writer-lock attempt is authoritative.

The current background-compute producer queues every session whose
`compute_version` is stale. It does not exclude active ProcessRuns. The compute
worker reads SessionItems from TwiCC SQLite and has no coordination with Codex's
rollout writer lock. Migration must therefore run in the main process before a
session is queued to the compute worker, not inside the worker.

Required TwiCC migration gate
-----------------------------

Use one main-process, per-session asynchronous gate shared by these paths:

- legacy-to-paginated migration;
- Codex thread resume/start for an existing session;
- watcher ingestion and reset for that session.

The gate decides the in-process race:

1. A migration attempt acquires the gate.
2. It rechecks SessionMeta and the live-agent registry.
3. If a TwiCC agent is active, it releases the gate and defers migration.
4. Otherwise it records share timestamp anchors, pauses that session's watcher,
   and invokes Codex migration.
5. A message arriving after step 1 waits on the same gate. After successful
   rebuild, it resumes the now-paginated thread and is not lost.
6. A message/resume that acquired the gate first establishes Codex's live writer
   before releasing it. A later migration sees the active manager and defers;
   Codex's writer lock remains the final cross-process guard.

Do not hold the Codex manager's global lock for the full migration. That would
block unrelated sessions. The new gate must be keyed by session id.

The TwiCC gate must not be durable. It is an in-memory `asyncio.Lock` owned by
the main process. `async with` releases it on normal completion, errors, and
task cancellation. A TwiCC process restart or crash destroys it automatically.
An unlocked object left in the per-session lock map is harmless and can be
removed when it has no owner or waiter.

The gate must not be held while waiting for an active session to stop. It only
covers an actual migration attempt and the structural TwiCC rebuild for that
session. If the live-agent pre-check or Codex returns `skipped_busy`, TwiCC
releases the gate immediately, records the session id in an in-memory deferred
set, and continues with the next compute candidate.

The gate belongs in the Codex main-process migration coordinator. It does not
belong in the background compute subprocess, SQLite, or the global compute
queue. The Codex agent resume/send path and Codex watcher consult the same gate
only to avoid entering a session while its migration and structural rebuild are
actually running. Unrelated sessions use different gates and continue.

Codex's file-lock paths can remain on disk after a crash, but the path is not
the lock state. The operating system releases the advisory lock when the owning
process exits. TwiCC must never decide that a thread is busy merely because the
`.lock` file exists, and it must never delete that file to force an unlock.

Crash recovery must depend on durable migration state, not on a durable lock:

- leave `compute_version` stale until the source migration and TwiCC rebuild
  both finish;
- retain the temporary frozen-share timestamp anchor in existing
  `Share.options` until its new line number is committed;
- on restart, inspect the source format and stale compute version, then resume
  migration recovery or rebuild from byte zero;
- let Codex recover its own interrupted publication through its `.pending`
  migration journal.

If TwiCC dies while a spawned Codex migration process remains alive, that
process continues to own Codex's writer lock. The restarted TwiCC instance gets
`skipped_busy` and retries later. If TwiCC enforces a migration timeout, it must
terminate the child and wait for child exit before releasing its local gate.

Watcher coordination must also cover a watcher callback that started before the
migration flag was set. The migration waits for that callback, suppresses rename
events while replacing/rebuilding, then resets the session to the migrated
file's new EOF/line state before releasing the gate.

Busy and retry policy
---------------------

`skipped_busy` is an expected deferred result, not a migration failure:

- do not purge SessionItems;
- do not change snapshot-share anchors;
- do not advance the session compute version;
- release the per-session gate;
- retry when the TwiCC agent reaches DEAD, with a periodic retry as the fallback
  for external Codex writers that TwiCC cannot observe directly.

The startup background-compute task may remain alive while a long-running
session is stale, but it does not block the app or other sessions. Its CPU
worker sleeps on the command queue until a candidate becomes ready.

Current TwiCC behavior must not be confused with the required deferred retry:

- the provider-agnostic background compute queues every stale session at once;
- for both Claude Code and Codex, `apply_session_complete` discards a result if
  the watcher advanced `last_offset` while the worker computed it;
- that discard leaves `compute_version` stale, but the current code retries it
  only on the next provider/TwiCC start, not later in the same run;
- Codex initial sync has a separate multi-sweep retry for subagents whose parent
  is not yet resolvable. That mechanism is Codex-specific and is not the
  background compute scheduler.

The rollout migration therefore extends the existing Codex compute coordinator:

1. The startup producer tries to prepare each Codex session before queueing its
   metadata compute.
2. A ready or successfully migrated session is queued normally.
3. A busy session is added to the deferred set, and the producer immediately
   examines the next session.
4. After each applied result, the producer rescans stale sessions by descending
   `mtime`, so a newly available recent session becomes the next candidate.
5. If only busy sessions remain, the same producer waits for agent `DEAD` or a
   periodic timeout while leaving the worker idle.
6. Once no stale session remains, it sends the existing `None` stop command and
   completes the run normally.
7. A restart needs no durable deferred queue: the legacy source format and stale
   `compute_version` make the session discoverable again.

If migration acquired Codex's writer lock, TwiCC-owned incoming messages wait on
the local gate instead of reaching Codex and receiving a conflict. An external
Codex client can still race and receive Codex's conflict, but it cannot corrupt
the rollout.

TwiCC currently creates legacy-shaped database content itself
-------------------------------------------------------------

Migrating every source JSONL does not by itself remove every legacy parser.
`CodexSessionCompute._transform_inline_provider()` rewrites selected raw lines
inside `SessionItem.content` into private legacy-shaped records:

- a visible Goal context becomes `event_msg.user_message` with
  `message="/goal ..."`;
- injected `/goal clear`, `/compact`, and bare `/plan` commands become
  `event_msg.user_message`;
- a Plan-mode `<proposed_plan>` raw assistant response becomes
  `event_msg.agent_message`;
- `/plan <prompt>` restoration mutates the flat legacy `payload.message`;
- screenshot-tag substitution mutates the flat legacy assistant
  `payload.message`.

These transformed rows keep their raw source payload under
`twiccOriginalContent`. Several backward DB scans also search for the private
legacy shape, especially the bare `/plan` marker.

If the target architecture has only one paginated content reader, these
private transformations must change too. They can either:

- emit TwiCC-private canonical `item_completed/UserMessage` and
  `item_completed/AgentMessage` shapes; or
- stop rewriting raw JSON and store presentation overrides in separate
  provider-neutral fields.

Keeping the current transformations would force the new code to parse both
paginated source lines and TwiCC-created legacy-shaped database lines.

Codex itself contains a useful compatibility specification
-----------------------------------------------------------

`codex-rs/protocol/src/legacy_events.rs` converts canonical TurnItems back to
legacy EventMsg values for old clients. This is independent confirmation of
the field mapping above.

Useful compatibility rules from that file:

- `UserMessageItem.message()` concatenates every text input and ignores
  non-text inputs;
- image, local-image, audio, and local-audio entries are rebuilt as the old
  sibling arrays;
- text-element byte ranges are rebased across concatenated text chunks;
- each `AgentMessage` text entry becomes one legacy agent-message event;
- `FileChange.status == completed` is the replacement for legacy
  `success == true`;
- missing optional FileChange stdout/stderr become empty legacy strings;
- MCP `arguments == null` becomes a missing legacy invocation argument;
- MCP result wins over MCP error if both exist;
- a canonical MCP item without result, error, or duration cannot synthesize a
  complete legacy end event;
- `SubAgentActivity.completed_at_ms` becomes legacy `occurred_at_ms`;
- an extension-owned image-generation item is not handled by the generic
  `TurnItem.as_legacy_events()` branch, so TwiCC must match both
  `type == "Extension"` and `kind == "image_gen.generation"` directly.

The TwiCC parser can reuse these semantics without retaining support for the
old physical JSONL event names.

Canonical user inputs that have no legacy sibling array
--------------------------------------------------------

New paginated `UserMessage` items can also contain:

- `{"type":"skill","name":...,"path":...}`;
- `{"type":"mention","name":...,"path":...}`.

The Codex legacy compatibility adapter ignores these entries when it rebuilds
the flat legacy message. Migrated old sessions cannot contain them because the
old user-message event had no equivalent fields.

For strict parity, TwiCC may ignore them. For complete support of new Codex
sessions, the canonical parser should preserve or render them intentionally.

How Codex 0.151 selects paginated history
-----------------------------------------

The protocol enum still defaults to legacy. The 0.151 app-server now overrides
that default for non-ephemeral new threads when the local thread store supports
paginated history lists. This is why a normal new SDK-created session becomes
paginated even though the SDK request shape did not change.

The optional background migration feature remains disabled by default at the
0.151 tag. It does not need to migrate all old sessions automatically.

Codex exposes a manual per-session command:

`codex migrate-rollouts --apply --thread <full-thread-id> --json`

Multiple `--thread` flags are accepted. Without `--apply`, the command is a dry
run. Per-thread statuses include `eligible`, `migrated`, `already_paginated`,
`skipped_empty`, `skipped_busy`, and `failed`.

The apply path requires the thread to exist in Codex's SQLite metadata. It
takes maintenance and writer locks, stages a canonical JSONL, projects that
staged file into SQLite, verifies offsets and ordinals, then publishes the
replacement. A TwiCC coordinator must still prevent its own watcher from
reading the replacement as an append.

Exact migration timestamps and synthesized identifiers
-------------------------------------------------------

For a converted completed item, Codex:

- preserves the old line's top-level RFC3339 `timestamp`;
- derives outer `payload.completed_at_ms` from that timestamp;
- sets outer `payload.started_at_ms` to null/missing;
- preserves an explicit old turn id when the old event carried one;
- otherwise uses the active turn or synthesizes `rollout-<source-index>`;
- synthesizes missing item ids as `item-1`, `item-2`, and so on.

TwiCC currently stores the top-level timestamp as `SessionItem.timestamp`.
That behavior can remain unchanged. TwiCC does not need the synthesized turn
or item ids for user/assistant presentation, but it must preserve call-derived
item ids for FileChange, MCP, and SubAgentActivity linking.

Records with no lossless migration replacement
----------------------------------------------

Codex explicitly optimizes migration for model-visible conversation, not for
byte-for-byte preservation of every old record.

### Empty legacy assistant message

An `event_msg.agent_message` whose `payload.message` is empty is dropped. No
canonical `AgentMessage` item is emitted for it.

TwiCC currently classifies that old line as an assistant message. The frontend
can show an “empty response” notice for it. That notice disappears after
migration unless TwiCC derives the empty-turn state from the surviving turn
boundary records or preserves a private marker before the rebuild.

### Legacy record larger than 16 MiB

The migration reader discards any complete JSONL record larger than
`16 * 1024 * 1024` bytes. It then resumes at the next newline.

This can remove a large raw `response_item.function_call_output` or
`custom_tool_call_output`. TwiCC currently uses those lines for tool output,
shell transcripts, screenshot discovery, errors, and ToolResultLink rows.
Codex does not synthesize a canonical FunctionCallOutput from such a discarded
legacy raw line. This is real data loss for that exceptional session.

If TwiCC must preserve it, the migration coordinator must detect oversized
source lines before invoking Codex and copy their useful content into separate
TwiCC-owned storage keyed by a semantic call id. Keeping the old SessionItem
row under its old line number is unsafe.

### Malformed, partial, and retired records

The migration skips malformed complete lines, an incomplete trailing line,
blank lines, and known retired records. TwiCC currently imports non-empty raw
lines even when their JSON is invalid, but compute cannot derive normal
metadata from them. The practical loss is debug-only raw history, not a mapped
conversation or tool feature.

An unsupported parsed `ResponseItem::Other` is different: canonicalization
fails, so Codex does not publish a partially migrated JSONL.

Implementation boundary derived from this audit
-----------------------------------------------

Supporting only paginated source history requires five coordinated changes:

1. Detect a legacy SessionMeta before normal compute and invoke Codex's
   per-thread migration while TwiCC's watcher cannot ingest that file.
2. Replace the six legacy event readers with the six canonical item readers in
   the backend and frontend call sites listed above.
3. Convert TwiCC's private legacy-shaped DB transformations to canonical or
   provider-neutral shapes.
4. Purge and rebuild every line-derived row and index from byte zero after the
   source rewrite. Clear ToolResultLink and AgentLink rows explicitly.
5. Apply the accepted policies: discard TwiCC-only `original_files`, and remap
   each snapshot `frozen_at_line` automatically through its timestamp anchor.

The parser change alone is insufficient. The migration rewrites physical line
positions, and TwiCC stores those positions in several durable structures.

Verification status and required regression coverage
----------------------------------------------------

Current TwiCC baseline tests were run for the affected semantic paths:

`uv run pytest tests/test_user_messages_endpoint.py tests/test_codex_code_mode.py tests/test_codex_subagent_links.py tests/test_plan_docs_providers.py tests/test_codex_hardcoded_commands.py tests/test_insert_screenshot.py -q`

Result: 207 tests passed. These tests confirm the current legacy behavior that
the canonical implementation must preserve. Most fixtures still use legacy end
events, so passing today does not prove paginated support.

The implementation needs canonical fixtures and assertions for at least:

- user text, multiple text chunks, image-only input, data-URL image, and local
  image count;
- assistant text, multiple Text chunks, empty migrated assistant loss, and
  screenshot substitution;
- direct and nested FileChange linking, success, failed, declined, path
  extraction, plan-doc updates, line statistics, diff rendering, and
  `original_files` transfer/fallback;
- direct and nested McpToolCall linking, success, transport error, and
  `result.isError == true` with a retained result body;
- started SubAgentActivity in both batch and live paths, DB backward lookup,
  AgentLink creation, and later FINAL_ANSWER rebinding;
- successful and failed image-generation Extension rendering, including
  camelCase fields and failure metadata;
- deliberate de-duplication of raw ResponseItems versus canonical completed
  items for reasoning, function outputs, plans, patches, MCP, and images;
- migration rewrite handling: reset to byte zero, rebuilt line-derived state,
  and no stale frontend line-keyed state;
- loss of historical `original_files` with correct hunk fallback;
- automatic snapshot timestamp remapping, including same-timestamp output lines,
  unchanged public tokens, and no owner action.

Codex's Rust migration tests and source were inspected at tag
`rust-v0.151.0`. The local environment has no `cargo` executable, so the Rust
tests could not be executed here. The field mappings are verified directly
against `rollout_migration/legacy_event.rs`, `canonicalizer.rs`, the serde
types, the persistence policy, and Codex's reverse compatibility adapter.

Authoritative source index
--------------------------

Codex 0.151 sources:

- `codex-rs/rollout/src/policy.rs`: complete persistence allow/deny policy for
  legacy and paginated history;
- `codex-rs/thread-store/src/local/rollout_migration/legacy_event.rs`: frozen
  legacy event to canonical TurnItem field mapping;
- `codex-rs/thread-store/src/local/rollout_migration/canonicalizer.rs`:
  wrapper creation, ordinals, turns, synthesized ids, and copied records;
- `codex-rs/thread-store/src/local/rollout_migration/line_parser.rs`: malformed,
  partial, retired, and oversized record handling;
- `codex-rs/thread-store/src/local/rollout_migration.rs`: staged projection,
  verification, and atomic publish workflow;
- `codex-rs/thread-store/src/local/rollout_migration_tests.rs`: migration and
  projection regression cases;
- `codex-rs/protocol/src/items.rs`: canonical TurnItem serialization and field
  definitions;
- `codex-rs/protocol/src/user_input.rs`: canonical UserMessage content variants;
- `codex-rs/ext/items/src/lib.rs` and `image_generation.rs`: flattened
  Extension/image-generation kind and camelCase fields;
- `codex-rs/protocol/src/legacy_events.rs`: canonical-to-legacy compatibility
  semantics, useful as an independent mapping specification;
- `codex-rs/cli/src/migrate_rollouts.rs`: per-thread command, dry-run/apply,
  status, and JSON output.

Primary TwiCC sources:

- `src/twicc/providers/codex/compute.py`: classification, content analysis,
  links, DB backward lookups, private transforms, costs, tasks, and documents;
- `src/twicc/providers/codex/helpers.py`: search/message extraction, linked
  result payloads, and live stream UUID enrichment;
- `src/twicc/providers/codex/initial_sync.py`: SessionMeta extraction;
- `src/twicc/providers/codex/sessions_watcher.py` and
  `src/twicc/providers/sessions_watcher.py`: live append ingestion and derived
  broadcasts;
- `src/twicc/sync_helpers.py`: offset-based append reader;
- `src/twicc/providers/compute_base.py`: batch/live hook orchestration and link
  creation;
- `src/twicc/core/session_queries.py`, `src/twicc/search.py`,
  `src/twicc/search_indexing_task.py`, `src/twicc/views.py`, and
  `src/twicc/cli/session.py`: indirect DB consumers;
- `frontend/src/providers/codex/helpers.js` and `toolHelpers.js`: message and
  result normalization for frontend consumers;
- `frontend/src/components/session/detail/items/codex/Message.vue`,
  `ApplyPatchContent.vue`, `ImageGeneration.vue`, and
  `PlanImplementationBody.vue`: direct legacy field readers;
- `frontend/src/stores/data.js` and
  `frontend/src/components/session/detail/SessionItemsList.vue`: line-number
  identity, optimistic reconciliation, grouping, and virtual scrolling.
