Metadata-Version: 2.5
Name: shidoshi
Version: 0.0.8
Summary: An opinionated way to augment Jupyter Lab for iterative work
Project-URL: Homepage, https://github.com/tdchaitanya/shidoshi
Project-URL: Repository, https://github.com/tdchaitanya/shidoshi
Project-URL: Issues, https://github.com/tdchaitanya/shidoshi/issues
Author-email: Chaitanya <tdchaitanya@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-assistant,jupyter,jupyterlab,llm,openai
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Jupyter
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.13
Requires-Dist: deepagents<0.8,>=0.7.5
Requires-Dist: ipylab>=0.6
Requires-Dist: ipynbname>=2025.8.0.0
Requires-Dist: ipython>=8.39.0
Requires-Dist: langchain-openai>=1.5.0
Requires-Dist: langchain-openrouter>=0.2.7
Requires-Dist: openai>=2.41.1
Requires-Dist: platformdirs>=4.10.0
Requires-Dist: pydantic>=2.12.5
Requires-Dist: tomlkit>=0.15.1
Requires-Dist: trio>=0.25
Requires-Dist: typing-extensions>=4.0
Provides-Extra: notebook
Requires-Dist: ipywidgets>=8.0; extra == 'notebook'
Description-Content-Type: text/markdown

# shidoshi

[![PyPI](https://img.shields.io/pypi/v/shidoshi.svg)](https://pypi.org/project/shidoshi/)
[![CI](https://github.com/tdchaitanya/shidoshi/actions/workflows/test.yml/badge.svg)](https://github.com/tdchaitanya/shidoshi/actions/workflows/test.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

An opinionated way to augment Jupyter Lab for iterative work.

shidoshi adds `%ask` / `%%ask` magics to Jupyter that let you talk to an LLM
from inside a notebook — using the notebook itself, in order, as the
conversation history. No separate chat pane, no copy-pasting context: your
code cells, their outputs, and your notes are the context.

## Install

Requires Python ≥3.13 and JupyterLab. Set an API key before use:

```bash
export OPENAI_API_KEY=sk-...       # for the openai provider (default)
export OPENAI_BASE_URL=...         # optional, e.g. to point at a proxy
export OPENROUTER_API_KEY=...      # for the openrouter provider
```

### Installing across Jupyter environments

`%load_ext shidoshi` runs `import shidoshi` **inside the running kernel
process**. That means shidoshi has to be installed into whichever Python
environment the kernel you're using actually runs in. It ships a small
`shidoshi` command for setup, but the library itself is not a standalone
tool — so `uvx` / `uv tool install` (which run a tool in an isolated
subprocess, separate from any kernel) don't apply here.

- **Per-project venv with its own JupyterLab** (e.g. a `uv`-managed project):
  add shidoshi as a normal dependency of that project.

  ```bash
  uv add shidoshi
  # or: pip install shidoshi
  ```

- **One shared JupyterLab, many kernels** (each notebook's kernel points at a
  different project venv registered via `ipykernel install`): install
  shidoshi into *each* kernel's venv. Installing it only where JupyterLab
  itself lives will not make it importable from other kernels.

  ```bash
  # inside the venv backing a given kernel
  uv add shidoshi
  # or: pip install shidoshi
  ```

### Skipping `%load_ext` — the shidoshi kernel

To avoid typing `%load_ext shidoshi` in every notebook, register a kernel that
loads it for you:

```bash
shidoshi install-kernel
```

Pick **Python 3 (shidoshi)** from the Jupyter kernel list and the magics are
already there. Nothing else changes: it is a stock Python kernel running this
environment's interpreter — your imports, variables, and debugger all work
exactly as before. The generated `kernel.json` just appends
`--IPKernelApp.extensions=shidoshi` to the normal `ipykernel_launcher`
command, with an absolute path to this environment's Python.

With no location flag, this installs into your **per-user** kernel directory
— the only location every Jupyter on the machine can find, regardless of
which environment actually launches it. After installing, the command runs
`jupyter --paths` for you and prints a warning if the kernel it just wrote
won't actually be visible to the `jupyter` on your `PATH` — worth reading if
you pass `--sys-prefix` or `--prefix` and the kernel doesn't show up in the
launcher.

Useful flags:

| flag | effect |
| --- | --- |
| *(none)* | install into your per-user kernel directory — visible to any Jupyter on this machine. **Default.** |
| `--sys-prefix` | install into the active venv — only visible to a Jupyter launched from this same environment |
| `--prefix PATH` | install into an explicit prefix |
| `--system` | install system-wide, for every user on the machine (usually needs root) |
| `--name` / `--display-name` | override the ids — use a distinct `--name` per environment if you register more than one |
| `--env KEY=VALUE` | set an environment variable for the kernel process (repeatable) |
| `--force` | replace an existing kernelspec of the same name (logos in it are kept) |

Installing is refused if a kernelspec of that name already exists, so it won't
quietly replace one you made by hand. Register one per environment with a
distinct `--name`.

Remove it with `jupyter kernelspec remove shidoshi`.

#### With uv

Add shidoshi to the project, then register the kernel from inside it. **Which
location flag you need depends on where JupyterLab itself runs from**, because
Jupyter only searches its own `sys.prefix`, your user directory, and the system
directory:

```bash
uv add shidoshi

# A: JupyterLab in an ephemeral env (uv's default suggestion).
#    Its sys.prefix is a uv cache dir, so --sys-prefix would be invisible.
uv run shidoshi install-kernel --user
uv run --with jupyter jupyter lab

# B: JupyterLab as a project dependency — sys.prefix *is* the project venv.
uv add --dev jupyterlab
uv run shidoshi install-kernel --sys-prefix
uv run jupyter lab
```

B keeps the kernel scoped to the project and disappears with the venv; A is
the one that works with `uv run --with jupyter` — and matches `install-kernel`'s
default, so `uv run shidoshi install-kernel` (no flag) does the same thing.
If a freshly installed kernel doesn't show up in the launcher, run
`jupyter kernelspec list` **the same way you start Lab** — that prints exactly
the directories being searched, or trust `install-kernel`'s own post-install
warning, which runs that same check for you.

Either way this replaces the kernel step in
[uv's Jupyter guide](https://docs.astral.sh/uv/guides/integration/jupyter/) —
you don't need `uv run ipython kernel install --env VIRTUAL_ENV ...` as well,
because `install-kernel` records `VIRTUAL_ENV` for you when it detects a venv.

That variable matters more than it looks. `uv pip install` resolves its target
from `VIRTUAL_ENV` (falling back to `CONDA_PREFIX`, then a base interpreter) —
never from the kernel that's running. So if you start Jupyter from a
conda-activated shell, a kernel without `VIRTUAL_ENV` will `import` from your
project venv while `!uv pip install` quietly installs into your conda base.
Pinning it keeps both views on the same environment.

Two related notes:

- `!uv add` was always safe — it finds the project by walking up for
  `pyproject.toml`, so it ignores `VIRTUAL_ENV` and targets the project venv
  either way.
- `%pip install` needs `uv venv --seed`; uv venvs have no `pip` in them by
  default. Prefer `!uv add`.

Pass `--env` to set anything else the kernel should launch with (repeatable),
including an override for `VIRTUAL_ENV`:

```bash
uv run shidoshi install-kernel --sys-prefix --env OPENAI_BASE_URL=http://127.0.0.1:18080/v1
```

Because the spec pins an absolute interpreter path, it can only ever start the
environment shidoshi is installed in. That is the advantage over the
`ipython_config.py` route below: `~/.ipython` is shared by *every* Python
environment under your `$HOME`, so putting the extension there makes every
kernel on the machine try to import shidoshi, including ones that don't have
it.

<details>
<summary>Auto-loading via ipython_config.py instead</summary>

Add to `~/.ipython/profile_default/ipython_config.py` (create it first with
`ipython profile create`):

```python
c.InteractiveShellApp.extensions = ["shidoshi"]
```

Only safe if shidoshi is installed in **every** environment you use for
Jupyter on that machine. To scope it, create a named profile
(`ipython profile create shidoshi`), put the `extensions` line in that
profile, and add `"--profile=shidoshi"` to the relevant kernel's `argv`.

</details>

## Quickstart

```
%load_ext shidoshi
```

(skip this line if you're on the **Python 3 (shidoshi)** kernel)

```
%%ask
What does the `history.build_history` function in this file do?
```

The response streams into the cell's output as Markdown.

## Magics reference

- **`%ask <prompt>`** — line magic for a one-line prompt.
  - Prefix with `model|` or `provider:model|` to override the configured
    default model for just this call, e.g. `%ask openrouter:openai/gpt-4o|summarize this`.
  - Add `--no-search` to make web search unavailable for this call, or
    `--search` to override a disabled profile. With `model | prompt`, put
    flags before `|`; prompt text after it is preserved verbatim.
  - Add `--debug` before `|` to also show the full request/response payload.
- **`%%ask [model]`** — cell magic; the whole cell body is the prompt
  (multi-line is fine, and it can reference images via Markdown
  `![]()`/`<img>` syntax or bare local file paths — they're inlined as
  base64). An optional model name on the magic line overrides the default
  for this call. Also supports `--search`, `--no-search`, and `--debug`.
- **`%%skip`** — runs the cell normally, but the cell is left out of the
  context sent to the model entirely. Use it for scratch or exploratory
  cells you don't want the model to see.
- **`%%pin`** — runs the cell normally; its content and output are *always*
  included in context and are exempt from the auto-trim behavior below. Use
  it to protect a fact, constant, or definition you don't want dropped over
  a long session.
- **`%%agent [model]`** — a multi-step, tool-using agent instead of a
  one-shot answer. See below.
- **`%agent_resume approve|reject|edit`** — answers a `%%agent` turn that
  paused for approval (only relevant if `agent.approvals` is configured —
  see [Configuration](#configuration)).
- **`%shidoshi config ...`** — inspect and edit shidoshi's own config from
  inside a notebook cell. See [Configuration](#configuration).

## `%%agent` — the agentic magic

Where `%%ask` answers once from what the notebook *shows*, `%%agent` can take
several steps and look at what the kernel actually *holds* — including
variables no cell output ever displayed. It runs
[deepagents](https://docs.langchain.com/oss/python/deepagents/) in-process, as
a backend parallel to `%%ask`'s; `%%ask` is unchanged.

```
%%agent
Which of my dataframes has missing values, and where?
```

The answer renders as Markdown with the agent's steps in a collapsed
**🧠 Agent steps** panel above it.

### Tool rungs

`--tools` chooses what the agent is allowed to do. **The default is `actor`:
model-written code runs in your live namespace, with no confirmation step.**
That is the reason `%%agent` exists rather than being a slower `%%ask`, so it
does not sit behind a flag — but know that it is what a bare `%%agent` cell
does.

| rung | what it can do |
|---|---|
| `none` | no notebook/kernel tools; configured provider-hosted search remains separate |
| `observer` | list and inspect kernel variables |
| `proposer` | the above, plus propose a cell for you to run |
| `actor` *(default)* | the above, plus **execute code in your kernel directly** |

`--no-tools` is the way back down — short for `--tools none`, for when you want
the conversation without the hands. `--tools proposer` is the middle ground: the
agent hands you a cell and *your Run button is the approval step*, with no
separate permission prompt to click.

`--no-tools` does not mean offline: web search is provider-hosted rather than a
kernel tool. Combine `--no-tools --no-search` when you want the model to use
only the notebook conversation and its existing knowledge.

Set a different default in config with `agent.permission_rung` —
`permission_rung = "proposer"` restores the gated behavior for every cell.
Go a step further and require an explicit approval before `run_code` itself
even runs — see [Configuration](#configuration).

These rungs also govern the opt-in project filesystem and host shell; see
[Letting `%%agent` work with real files](#letting-agent-work-with-real-files).

### Memory

Each notebook gets one conversation thread, seeded once from the cells above
the first `%%agent` call. Later cells continue that conversation rather than
rebuilding context each time.

The thread lasts as long as the kernel and no longer. This is deliberate: what
the agent remembers is largely *kernel state* — variables it listed, values it
read — and a restart destroys exactly that. A conversation that outlived the
kernel would keep tool results reporting variables that no longer exist, worded
as fact. So a restart clears the thread and the notebook is read again as it
currently stands, which is what rerunning it from the top means anyway.

Use `--fresh` to opt out of the thread entirely and get `%%ask`-style
behavior — context rebuilt from cells, nothing remembered — or `--thread NAME`
to keep a side conversation separate.

### Inspecting a turn

`--debug` works as it does for `%%ask`, adapted to a graph that runs more than
once. Above the answer you get the request panel — provider and model, the
composed system prompt, every message the model will see, and the tools this
rung binds — followed by a live panel of raw LangGraph events as the turn runs.
Messages already in the thread are labelled `from thread`, because on a
continued conversation only your new prompt is passed in; the rest comes from
the checkpoint.

The stream panel is LangGraph's fullest per-step view — `task`, `task_result`
and `checkpoint` events with step numbers, the middleware nodes that a plain
update stream never names, and a per-task error when one fails. Failed steps are
flagged in the summary line so you don't have to expand them to find the one
that broke. A running token count sits above it, for both providers.

Token deltas are collapsed into a per-node count, so what you read is the node
transitions — `model` → `tools` → `model` — rather than several hundred
one-token lines.

An unrecognised flag is an error rather than a no-op, so a typo'd `--tools`
tells you instead of quietly running with the default.

### Getting help

`%ask --help` and `%agent --help` print the flags, and for `%%agent` the rung
table. Use the **single-`%` line form**: IPython rejects a cell magic whose body
is empty before the magic itself runs, so `%%agent --help` on its own can never
reach us.

`%agent [flags] [model |] your request` is also a one-line shorthand for
`%%agent`, matching `%ask`. With an explicit `model | request`, flags belong
before `|`; text after it is always treated as the request.

## Letting `%%agent` work with real files

By default, Deep Agents' files live only in the in-memory conversation state.
They are useful as temporary notes, but they are not your project files. You
can opt into a rooted filesystem backend when you want `%%agent` to inspect a
project beside the notebook, propose changes to it, or run approved shell
commands there.

This is an explicit per-configuration capability. Existing notebooks keep the
`state` backend and receive no host-filesystem access until you select
`filesystem`.

### Start read-only

Add this to project, notebook, or global configuration:

```toml
[agent.harness.backend]
type = "filesystem"
root_dir = "."
```

Save the notebook, then start with an observer turn:

```text
%%agent --tools observer
Summarize the Python package beside this notebook. Read files only.
```

The notebook must be running through a real JupyterLab or Jupyter Notebook
server. Shidoshi resolves the workspace from the server's notebook path; an
unsaved notebook or a standalone/headless kernel has no trustworthy path and
therefore fails closed. The configured `root_dir` must already exist, must be
relative to the notebook's directory, and cannot contain `..`.

For example, given `/work/analysis.ipynb`:

| `root_dir` | workspace exposed as `/` |
|---|---|
| `"."` | `/work` |
| `"project"` | `/work/project` |
| `"../project"` | rejected |
| `"/work/project"` | rejected; use a relative path |

The paths seen by the model are virtual POSIX paths. `/src/app.py` means
`src/app.py` below the resolved workspace, not the host machine's `/src`.
Containment is enforced again at the backend boundary after resolving
symlinks; changing path spelling cannot escape the workspace.

### Tools and permission rungs

The filesystem capability follows the same `--tools` rung selected for the
rest of `%%agent`:

| rung | real-filesystem tools |
|---|---|
| `none` / `--no-tools` | none |
| `observer` | `ls`, `read_file`, `glob`, `grep` |
| `proposer` | the same read/search tools; project writes remain unavailable |
| `actor` | read/search, plus `write_file` and `edit_file` when writes are set to `ask`; optionally `execute` |

This table is additive to the kernel tools described above. In particular,
`actor` can still run Python in the live kernel unless you separately configure
an approval for `run_code`. Provider-hosted web search is also separate and is
controlled by `--search` / `--no-search`.

The active notebook, Git metadata, `.env` files, and Shidoshi's own
runtime storage are always protected. Add project-specific virtual glob
patterns with `deny_paths`:

```toml
[agent.harness.backend]
type = "filesystem"
root_dir = "project"
deny_paths = ["/private/**", "/data/raw/**", "/*.pem"]
```

Denied paths cannot be read, searched, written, or reached through a symlink.
Matching is deliberately case-insensitive so a protected `/.env` cannot be
addressed as `/.ENV` on a case-insensitive volume. Denies are filesystem-tool
boundaries, not shell sandbox rules; an approved shell command is unrestricted
host execution.

### Approved file changes

Workspace mutation supports two policies:

```toml
[agent.harness.backend.filesystem]
write = "ask"  # ask | deny
```

- `ask` exposes write/edit tools only at the `actor` rung. Every proposed
  mutation pauses before touching disk.
- `deny` removes write/edit tools entirely.

Autonomous filesystem writes are intentionally unsupported. With `ask`, the
notebook displays an approval card containing the virtual target and proposed
diff. You can approve once, reject with guidance, or edit the arguments. A
filesystem edit creates a new diff and requires a second, explicit approval;
the old card is retired and cannot approve the revised proposal.

Shidoshi snapshots the target when it shows the preview. If the path, file
type, existence, or contents change before approval, the action becomes stale
and a fresh preview must be approved. This prevents an approval from applying
to a different file state than the one you reviewed.

The widget is the normal interface in JupyterLab and Notebook 7. The equivalent
command fallback is:

```text
%agent_resume approve --action act-0123
%agent_resume reject use docs/example.toml instead --action act-4567
%agent_resume edit {"file_path":"/docs/example.toml","content":"..."} --action act-0123
%agent_resume continue
%agent_resume abandon
```

For one pending action, `--action` may be omitted. When several tool calls are
pending together, decide each stable action ID and then run `continue`; the
decisions are resumed in their original order. `abandon` cancels the whole
pending batch without advancing the agent. Another turn on that thread cannot
silently replace an unresolved batch—resolve it or abandon it first. A
different named thread remains independent.

Use `--thread NAME` on `%agent_resume` when the paused action belongs to a named
side thread. Approval must resume the same live thread and filesystem root that
created the proposal; moving the notebook or changing the root while paused
fails closed.

### Approved host shell

Shell execution is a separate, actor-only opt-in:

```toml
[agent.harness.backend.shell]
enabled = true
execute = "ask"
default_timeout_seconds = 300
max_timeout_seconds = 600
max_output_kb = 100
pass_env = ["PATH"]
```

`execute = "ask"` is the only supported execution policy: every command pauses
before it starts. The card shows the exact command, parsed compound segments,
risk indicators, real working directory, timeouts, output cap, and environment
policy. Editing a shell proposal is itself an approving decision; for a
multi-action batch it runs only after `continue`.

This shell is **not a sandbox**. It runs on the Jupyter host with the workspace
as its working directory and can use absolute paths, follow symlinks, access
the network, or modify files outside the workspace with the kernel process's
permissions. Filesystem `deny_paths` do not constrain it. Approve only commands
you would run yourself in a terminal.

The shell does not inherit the kernel environment. Shidoshi constructs a new
environment from the names in `pass_env`; missing names are simply absent.
Credential-shaped names containing `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, or
`CREDENTIAL` are rejected by configuration validation. Values are never shown
in configuration diagnostics or the approval card.

Timeouts are enforced outside model control. Both timeout settings accept
1–3600 seconds, `default_timeout_seconds` cannot exceed
`max_timeout_seconds`, and an individual command cannot request more than the
configured maximum. `max_output_kb` accepts 1–1024 KiB and truncates captured
output after the limit. The defaults are five minutes, ten minutes, and
100 KiB respectively.

### Scratch files and large tool results

Deep Agents may offload large intermediate tool results into private scratch
storage so the model context stays manageable. Scratch is not part of the
workspace: the model cannot list, search, edit, or create deliverables there.
It can only read an exact artifact path returned by the harness.

The default location is Shidoshi's per-user state directory, outside the
notebook project:

```toml
[agent.harness.backend.scratch]
location = "user_state"       # user_state | notebook_local
retention_days = 7            # 1..365
max_size_mb = 256             # 1..1024
```

Scratch is isolated by notebook and agent thread, created with user-only
permissions, capped by `max_size_mb`, and garbage-collected after the retention
window. Explicitly abandoning a pending thread attempts immediate cleanup.
The scratch host path is not exposed to the model.

If you need all runtime data beside the notebook, opt in explicitly:

```toml
[agent.harness.backend.scratch]
location = "notebook_local"
notebook_dirname = ".shidoshi"
retention_days = 7
max_size_mb = 256
```

Shidoshi protects that directory from filesystem tools but does not edit your
`.gitignore`. It prints a warning when the directory is not already ignored;
add it yourself if the project should not track runtime artifacts:

```gitignore
.shidoshi/
```

### Complete configuration

This example shows every filesystem-backend setting and its default:

```toml
[agent.harness.backend]
type = "state"                # state | filesystem
root_dir = "."                # existing path below the notebook directory
deny_paths = []                # virtual glob patterns, additive to fixed denies

[agent.harness.backend.scratch]
location = "user_state"       # user_state | notebook_local
notebook_dirname = ".shidoshi"
retention_days = 7             # 1..365
max_size_mb = 256              # 1..1024

[agent.harness.backend.filesystem]
write = "ask"                 # ask | deny

[agent.harness.backend.shell]
enabled = false
execute = "ask"
default_timeout_seconds = 300  # 1..3600
max_timeout_seconds = 600      # 1..3600; must be >= default
max_output_kb = 100            # 1..1024
pass_env = ["PATH"]
```

Settings are resolved per turn. The backend type, canonical root, rung, tool
inventory, denies, write/shell policy, scratch policy, and limits form part of
the compiled-agent cache and thread identity. If one changes during a live
thread, Shidoshi refuses to continue with mismatched assumptions; use
`%%agent --fresh` to start with the new configuration or revert the change.
`--debug` shows the resolved host root and effective policy. Normal notebook
output shows only a concise scope strip and virtual paths.

### Troubleshooting filesystem access

| symptom | what to check |
|---|---|
| “cannot identify this notebook” | Save the notebook and run it through a real Jupyter server. Headless kernel execution cannot supply notebook identity. |
| `root_dir` is rejected | Use an existing directory relative to the notebook; absolute paths and `..` are not allowed. |
| no filesystem tools appear | Confirm `type = "filesystem"` and use `observer`, `proposer`, or `actor` rather than `--no-tools`. Use `--debug` to inspect the exact tool inventory. |
| reads work but writes do not | Writes require the `actor` rung and `filesystem.write = "ask"`. |
| `execute` does not appear | Shell also requires `actor` and `shell.enabled = true`. |
| a second turn is refused | The thread has unresolved actions. Approve/reject and `continue`, or use `%agent_resume abandon`. |
| approval became stale | The target changed after its preview. Review the newly rendered card and approve again. |
| shell cannot find an executable | Add a non-secret variable such as `PATH` to `pass_env`; the kernel environment is not inherited. |
| notebook-local scratch warning | Add the configured scratch directory to `.gitignore`, or switch back to `user_state`. |
| config validation rejects an environment name | Credential-bearing environment variables are intentionally forbidden from `pass_env`. |

Use `shidoshi config show --sources` to confirm the effective merged values,
`shidoshi config validate` for schema errors, and `shidoshi config schema
--html` for the generated field-by-field reference.

## How context is built

Every prior cell in the notebook — up to the one you're currently running,
and accounting for kernel restarts — is turned into conversation history
automatically:

- **Markdown cells** become background text/image context (treated as notes
  or reference material, not instructions).
- **Regular code cells** appear as fenced code plus their text/image
  outputs.
- **Prior `%ask` / `%%ask` cells** become real user/assistant turns. Their
  responses are reused from a per-cell cache rather than re-sent, so
  replaying history doesn't resend answers the model already produced.
- **`%%skip` cells** are dropped entirely.
- **`%%pin` cells** are always kept.

## Automatic context-length handling

If a request is rejected for exceeding the model's context window, shidoshi
automatically retries, dropping the oldest trimmable history units first
(markdown cells, then plain code cells, then whole ask+response pairs —
`%%pin` cells are never dropped), up to 20 times. A banner reports how many
cells were dropped so you know context shrank.

## Providers

- **`openai`** (default) — uses the OpenAI Responses API.
- **`openrouter`** — uses OpenRouter's chat-completions API; select it with
  the `provider:model` prefix, e.g. `openrouter:anthropic/claude-3.5-sonnet`.

OpenAI `%ask` and `%%agent` always use Responses, even when web search is off.
An OpenAI-compatible endpoint therefore needs to implement `/responses`;
Shidoshi rejects `providers.openai.api_mode = "chat_completions"` instead of
silently changing protocols. OpenRouter continues to use Chat Completions.

## Web search

Provider-hosted web search is available by default to `%ask`, `%%ask`, and
`%%agent`. “Available” does not mean every prompt is searched: the model decides
whether current information is needed. A search can add latency and cost, and a
model-generated query is sent to the selected provider/search engine.

### Turn search on or off

For one turn:

```text
%ask --no-search Explain this function from the notebook
%ask --search What changed in the latest Python release?

%%agent --no-tools --no-search
Review the notebook without inspecting the kernel or accessing the web.
```

`--no-search` removes the hosted search tool; it does not make the model call
local or offline. The prompt and notebook context are still sent to the chosen
LLM provider under its normal data policy.

The explicit `model | prompt` form keeps everything after `|` as prompt text:

```text
%ask --no-search openai:gpt-5.5 | Explain what --search means
%agent --search openrouter:z-ai/glm-5.3 | Find the current release notes
```

For the current kernel session:

```text
%shidoshi config session set tools.web_search.enabled false
%shidoshi config session set tools.web_search.enabled true
%shidoshi config session unset tools.web_search.enabled
```

Persistently, in a project, notebook, profile, or global config:

```toml
[tools.web_search]
enabled = false
```

Search changes are safe within an existing `%%agent` thread. Shidoshi rebuilds
the model binding and cache policy while retaining conversation memory. A turn
paused for approval resumes with the search policy it started with.

### What appears in notebook output

- `%ask`/`%%ask` show a collapsed tool-activity panel when search activity or
  citations are returned.
- `%%agent` shows hosted search calls in the **Agent steps** panel.
- Source URLs are clickable HTTP(S) links. Provider-supplied titles, queries,
  and URLs are escaped before rendering.
- A provider can confirm that search occurred without returning citation URLs;
  Shidoshi labels that state explicitly rather than inventing sources.
- No search panel means search was not observed. It may have been available but
  unused, or disabled for that turn; use `--debug` to inspect the effective plan.

`include_sources = true` asks OpenAI for the complete consulted source list.
OpenRouter has no equivalent request switch; routed models may emit inline
citations regardless of this setting. No provider guarantees citation URLs.

### Common policy

Portable settings apply to either provider:

```toml
[tools.web_search]
enabled = true
search_context_size = "medium"
allowed_domains = ["python.org"]
# excluded_domains = ["example.com"]
include_sources = true

[tools.web_search.location]
# country = "IN"       # two-letter ISO country code
# region = "Maharashtra"
# city = "Mumbai"
# timezone = "Asia/Kolkata"
```

Domains must be bare hostnames: omit schemes, paths, ports, credentials, and
wildcards. Up to 100 allowed and 100 excluded domains may be configured.

### OpenAI controls

```toml
[tools.web_search.openai]
external_web_access = true
return_token_budget = "default"
```

- `external_web_access = false` keeps the search tool available but limits it
  to OpenAI's cached/indexed content. It is not the same as `enabled = false`.
- `return_token_budget` is `"default"` or `"unlimited"`; it is intended for
  GPT-5+ reasoning search and may increase latency and cost.

### OpenRouter controls

```toml
[tools.web_search.openrouter]
engine = "exa"
mode = "instant"
max_results = 5
max_uses = 3
max_total_results = 15
# max_characters = 5000
```

Supported engines are `auto`, `native`, `exa`, `firecrawl`, `parallel`, and
`perplexity`. Modes are engine-specific:

| engine | supported modes |
|---|---|
| `exa` | `instant`, `fast`, `auto`, `deep-lite`, `deep`, `deep-reasoning` |
| `parallel` | `turbo`, `fast`, `basic`, `advanced` |
| all others | no `mode`; Shidoshi rejects one rather than silently ignoring it |

Some native/auto capabilities depend on the routed model. Shidoshi reports a
warning when it cannot prove that a configured filter or limit will be honored.
Known impossible combinations fail before a request is sent.

### Trying newly released provider fields

Provider APIs evolve faster than Shidoshi's named schema. Search-specific
escape hatches let you try a new field without waiting for a release:

```toml
[tools.web_search.openai.extra_tool_fields]
future_option = true

[tools.web_search.openrouter.extra_parameters]
future_option = true
```

Shidoshi validates that these are finite, JSON-compatible values, but cannot
validate their provider semantics. They cannot override a field Shidoshi
already manages. They are included in debug output, warnings, cache policy, and
model fingerprints. If a provider later removes or rejects a field, its API
error is surfaced; there is no reliable automatic capability negotiation.

These differ from `providers.openai.extra_body` and
`providers.openrouter.extra_body`, which are for new request-level fields
rather than fields inside the web-search tool.

`web_fetch` is a separate tool and remains off by default. Enable it with
`tools.web_fetch.enabled = true`.

## Debug mode

Add `--debug` to `%ask`, `%%ask`, or `%%agent` to render the effective search
plan, serialized provider tool, composed prompt, messages, and raw stream
events. This is the fastest way to distinguish “available but unused” from
“disabled” and to inspect provider extension fields.

## Search troubleshooting

| symptom | what to check |
|---|---|
| Search was not used | It is optional by default. Confirm `enabled: true` in `--debug`; make the need for current information explicit in the prompt. |
| Search ran but shows no sources | The provider confirmed usage without citation URLs. Try another routed model/engine or ask the answer to include visible links. |
| OpenRouter rejects `mode` | Set an explicit compatible engine: Exa modes and Parallel modes are not interchangeable. |
| OpenAI-compatible endpoint returns 404/unsupported endpoint | The OpenAI adapter requires `/responses`, including when search is disabled. Use OpenRouter for Chat-Completions-only services. |
| A config value seems ignored | Run `%shidoshi config explain tools.web_search.enabled` and retry with `--debug` to see the effective plan. |
| Need a turn without kernel tools or web search | Combine `--no-tools --no-search`; `--no-tools` alone leaves hosted search available. The LLM request itself is still remote. |

## Configuration

shidoshi is configured through layered TOML files — global, project, and
per-notebook — merged in that order, with a per-invocation override always
winning. Nothing lower is ever silently overridden without a trace:
`shidoshi config explain <key>` tells you exactly which file set the value
you're seeing.

### Where config lives

| scope | path | typical use |
|---|---|---|
| legacy global | `~/.shidoshi/config.toml` | old flat-key config, still read for backward compatibility |
| global | the platform user-config dir (`shidoshi config path` shows exactly where) | personal defaults across every project |
| project | nearest `.shidoshi/config.toml`, walking up from the notebook | team-shared, usually checked into the repo |
| notebook | `<notebook>.ipynb.shidoshi.toml` | one notebook's own overrides |

A project or notebook config isn't found by convention alone — write one:

```bash
shidoshi config init --project            # a short starter file
shidoshi config init --project --full     # every available setting, with defaults filled in
```

(or `%shidoshi config init --project` from a notebook cell — every
`shidoshi config ...` command below also works as `%shidoshi config ...`.)

### The essentials

```toml
config_version = 1

[defaults]
profile = "default"

[profiles.default]
model = "openai:gpt-5.5"

[profiles.default.generation]
reasoning = { effort = "low" }

[agent]
reasoning_effort = "none"  # fast default; raise for harder agent tasks
permission_rung = "actor"   # none | observer | proposer | actor

[tools.web_search]
# Available by default. Search may add latency/cost and sends a generated query
# to the configured provider. Add --no-search to a magic for one no-search turn.
enabled = true
```

The old flat keys (`default_model`, `agent_model`, `agent_tools`,
`reasoning_effort`, `ask_color`, `skip_color`) still work — shidoshi
migrates them automatically — but the nested form above is what
`shidoshi config init` writes now, and it's the only shape that can express
everything else on this page (profiles, tool policy, context policy,
approvals).

### Inspecting and editing config

```bash
shidoshi config path                # which files are actually in effect, in precedence order
shidoshi config show                # the fully merged, effective config
shidoshi config show --sources      # ...annotated with which file set each value
shidoshi config explain agent.permission_rung   # one setting's value and where it came from
shidoshi config validate            # parse/schema-check every layer, and flag pasted-in secrets
shidoshi config doctor              # validate, plus orphaned sidecars and missing credential env vars
shidoshi config doctor --online     # opt in to current provider metadata checks
shidoshi config set agent.permission_rung observer --project
shidoshi config unset agent.permission_rung --project
shidoshi config schema              # the complete field-by-field surface, as JSON Schema
shidoshi config schema --html       # ...or a single offline, browsable reference page
```

Each of these also runs as `%shidoshi config ...` inside a notebook cell,
with two differences: `edit` prints the resolved path instead of spawning
`$EDITOR` (a blocking subprocess doesn't belong in a kernel cell), and a
bare call with no `--global`/`--project`/`--notebook` flag automatically
includes the *current* notebook's own sidecar.

Kernel-local overrides are explicit and inspectable:

```text
%shidoshi config session set generation.temperature 0.2
%shidoshi config session show
%shidoshi config session unset generation.temperature
%shidoshi config session clear
```

For one call only, repeat `--set key=value` on `%ask`/`%%ask`/`%agent`/
`%%agent`, or use `--profile NAME`. Search also has the clearer `--search` and
`--no-search` aliases. The whole override is parsed and validated before a
request, checkpoint, or pending approval is touched.

Provider policy is translated separately for raw OpenAI Responses,
`ChatOpenAI`, raw OpenRouter, and `ChatOpenRouter`. In particular,
OpenRouter's default `data_collection = "deny"` and
`require_parameters = true` are sent on the request; custom OpenAI-compatible
endpoints must implement the Responses API for the OpenAI adapter. They can
choose `instruction_role` explicitly. Configuration display and diagnostics recursively redact
credentials, authorization headers, secret-like metadata, and URL query
values.

For newly released provider request parameters that Shidoshi does not yet
name, use the provider-scoped kwargs-style escape hatch:

```toml
[providers.openai.extra_body]
future_option = true

[providers.openrouter.extra_body]
future_option = true
```

The same table can be set under a profile's `providers` section. These values
reach both the raw and LangChain-backed request paths. Shidoshi rejects keys
that collide with request fields it owns (model, tools, reasoning, storage,
routing/privacy policy, and limits), recursively redacts the passthrough in
diagnostics, and includes it in model-build fingerprints. `extra_body` is for
JSON request-body parameters; arbitrary Python constructor objects remain
Python extension points because TOML cannot represent or validate them safely.

### What's configurable

Model and generation settings (per named profile — select one with
`defaults.profile`, or `ask.profile`/`agent.profile` to use a different one
for each magic), provider transport, which tools
`%ask`/`%%agent` may use (`tools.web_search`, `tools.web_fetch`,
`tools.create_cell`, kernel-tool output limits and secret redaction),
notebook context policy (what gets included, image handling, trim limits),
local response caching, and `%%agent`'s permission model — including,
if you want it, a real human-in-the-loop approval gate:

`context.include_markdown`, `include_code`, `include_outputs`, and
`include_images` control notebook history. The nested image policy
(`allow_local_files`, `allow_remote_urls`, and `max_file_bytes`) applies to
images in both the current prompt and notebook history.

```toml
[agent.approvals.run_code]
enabled = true
allowed_decisions = ["approve", "edit", "reject"]
```

With this set, an `actor`-rung `run_code` call pauses instead of running
immediately. Answer it with `%agent_resume approve`,
`%agent_resume reject [reason]`, or `%agent_resume edit <json args>`.

Real project files and approved host-shell execution are configured under
`agent.harness.backend`. They are opt-in and have their own rooted path,
protected-path, scratch, timeout, environment, and approval policies. See
[Letting `%%agent` work with real files](#letting-agent-work-with-real-files)
for the complete guide and copy-paste configuration.

`shidoshi config schema --html` is the complete reference — every setting,
its type, and its default, generated straight from the schema shidoshi
itself validates against, so it can never drift out of date.
Fields that cannot yet be enforced safely (for example durable agent
persistence or host-backed skill/memory paths) fail validation with a precise
reason; shidoshi does not accept them as inert promises.

## Development

```bash
uv sync
uv run pytest tests/unit tests/btp -v
```

Integration tests under `tests/integration/` require a live `OPENAI_API_KEY`
(or a proxy via `OPENAI_BASE_URL`) and are run with:

```bash
uv run pytest tests/integration/ -v -m integration
```

## License

Apache License 2.0 — see [LICENSE](LICENSE).
