Metadata-Version: 2.5
Name: shidoshi
Version: 0.0.4
Summary: An opinionated way to augment Jupyter Lab for iterative work
Project-URL: Homepage, https://github.com/tdchaitanya/shidoshi
Project-URL: Repository, https://github.com/tdchaitanya/shidoshi
Project-URL: Issues, https://github.com/tdchaitanya/shidoshi/issues
Author-email: Chaitanya <tdchaitanya@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-assistant,jupyter,jupyterlab,llm,openai
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Jupyter
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.13
Requires-Dist: ipylab>=0.6
Requires-Dist: ipynbname>=2025.8.0.0
Requires-Dist: ipython>=8.39.0
Requires-Dist: openai>=2.41.1
Requires-Dist: trio>=0.25
Requires-Dist: typing-extensions>=4.0
Provides-Extra: agent
Requires-Dist: deepagents>=0.7.5; extra == 'agent'
Requires-Dist: langchain-openai>=1.5.0; extra == 'agent'
Requires-Dist: langchain-openrouter>=0.2.7; extra == 'agent'
Provides-Extra: notebook
Requires-Dist: ipywidgets>=8.0; extra == 'notebook'
Description-Content-Type: text/markdown

# shidoshi

[![PyPI](https://img.shields.io/pypi/v/shidoshi.svg)](https://pypi.org/project/shidoshi/)
[![CI](https://github.com/tdchaitanya/shidoshi/actions/workflows/test.yml/badge.svg)](https://github.com/tdchaitanya/shidoshi/actions/workflows/test.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

An opinionated way to augment Jupyter Lab for iterative work.

shidoshi adds `%ask` / `%%ask` magics to Jupyter that let you talk to an LLM
from inside a notebook — using the notebook itself, in order, as the
conversation history. No separate chat pane, no copy-pasting context: your
code cells, their outputs, and your notes are the context.

## Install

Requires Python ≥3.13 and JupyterLab. Set an API key before use:

```bash
export OPENAI_API_KEY=sk-...       # for the openai provider (default)
export OPENAI_BASE_URL=...         # optional, e.g. to point at a proxy
export OPENROUTER_API_KEY=...      # for the openrouter provider
```

### Installing across Jupyter environments

`%load_ext shidoshi` runs `import shidoshi` **inside the running kernel
process**. That means shidoshi has to be installed into whichever Python
environment the kernel you're using actually runs in. It ships a small
`shidoshi` command for setup, but the library itself is not a standalone
tool — so `uvx` / `uv tool install` (which run a tool in an isolated
subprocess, separate from any kernel) don't apply here.

- **Per-project venv with its own JupyterLab** (e.g. a `uv`-managed project):
  add shidoshi as a normal dependency of that project.

  ```bash
  uv add shidoshi
  # or: pip install shidoshi
  ```

- **One shared JupyterLab, many kernels** (each notebook's kernel points at a
  different project venv registered via `ipykernel install`): install
  shidoshi into *each* kernel's venv. Installing it only where JupyterLab
  itself lives will not make it importable from other kernels.

  ```bash
  # inside the venv backing a given kernel
  uv add shidoshi
  # or: pip install shidoshi
  ```

### Skipping `%load_ext` — the shidoshi kernel

To avoid typing `%load_ext shidoshi` in every notebook, register a kernel that
loads it for you:

```bash
shidoshi install-kernel --sys-prefix
```

Pick **Python 3 (shidoshi)** from the Jupyter kernel list and the magics are
already there. Nothing else changes: it is a stock Python kernel running this
environment's interpreter — your imports, variables, and debugger all work
exactly as before. The generated `kernel.json` just appends
`--IPKernelApp.extensions=shidoshi` to the normal `ipykernel_launcher`
command, with an absolute path to this environment's Python.

Useful flags:

| flag | effect |
| --- | --- |
| `--sys-prefix` | install into the active venv (best for a project venv) |
| `--user` | install into your per-user kernel directory |
| `--prefix PATH` | install into an explicit prefix |
| `--name` / `--display-name` | override the ids — use a distinct `--name` per environment if you register more than one |
| `--env KEY=VALUE` | set an environment variable for the kernel process (repeatable) |
| `--force` | replace an existing kernelspec of the same name (logos in it are kept) |

Installing is refused if a kernelspec of that name already exists, so it won't
quietly replace one you made by hand. Register one per environment with a
distinct `--name`.

Remove it with `jupyter kernelspec remove shidoshi`.

#### With uv

Add shidoshi to the project, then register the kernel from inside it. **Which
location flag you need depends on where JupyterLab itself runs from**, because
Jupyter only searches its own `sys.prefix`, your user directory, and the system
directory:

```bash
uv add shidoshi

# A: JupyterLab in an ephemeral env (uv's default suggestion).
#    Its sys.prefix is a uv cache dir, so --sys-prefix would be invisible.
uv run shidoshi install-kernel --user
uv run --with jupyter jupyter lab

# B: JupyterLab as a project dependency — sys.prefix *is* the project venv.
uv add --dev jupyterlab
uv run shidoshi install-kernel --sys-prefix
uv run jupyter lab
```

B keeps the kernel scoped to the project and disappears with the venv; A is
the one that works with `uv run --with jupyter`. If a freshly installed kernel
doesn't show up in the launcher, run `jupyter kernelspec list` **the same way
you start Lab** — that prints exactly the directories being searched.

Either way this replaces the kernel step in
[uv's Jupyter guide](https://docs.astral.sh/uv/guides/integration/jupyter/) —
you don't need `uv run ipython kernel install --env VIRTUAL_ENV ...` as well,
because `install-kernel` records `VIRTUAL_ENV` for you when it detects a venv.

That variable matters more than it looks. `uv pip install` resolves its target
from `VIRTUAL_ENV` (falling back to `CONDA_PREFIX`, then a base interpreter) —
never from the kernel that's running. So if you start Jupyter from a
conda-activated shell, a kernel without `VIRTUAL_ENV` will `import` from your
project venv while `!uv pip install` quietly installs into your conda base.
Pinning it keeps both views on the same environment.

Two related notes:

- `!uv add` was always safe — it finds the project by walking up for
  `pyproject.toml`, so it ignores `VIRTUAL_ENV` and targets the project venv
  either way.
- `%pip install` needs `uv venv --seed`; uv venvs have no `pip` in them by
  default. Prefer `!uv add`.

Pass `--env` to set anything else the kernel should launch with (repeatable),
including an override for `VIRTUAL_ENV`:

```bash
uv run shidoshi install-kernel --sys-prefix --env OPENAI_BASE_URL=http://127.0.0.1:18080/v1
```

Because the spec pins an absolute interpreter path, it can only ever start the
environment shidoshi is installed in. That is the advantage over the
`ipython_config.py` route below: `~/.ipython` is shared by *every* Python
environment under your `$HOME`, so putting the extension there makes every
kernel on the machine try to import shidoshi, including ones that don't have
it.

<details>
<summary>Auto-loading via ipython_config.py instead</summary>

Add to `~/.ipython/profile_default/ipython_config.py` (create it first with
`ipython profile create`):

```python
c.InteractiveShellApp.extensions = ["shidoshi"]
```

Only safe if shidoshi is installed in **every** environment you use for
Jupyter on that machine. To scope it, create a named profile
(`ipython profile create shidoshi`), put the `extensions` line in that
profile, and add `"--profile=shidoshi"` to the relevant kernel's `argv`.

</details>

## Quickstart

```
%load_ext shidoshi
```

(skip this line if you're on the **Python 3 (shidoshi)** kernel)

```
%%ask
What does the `history.build_history` function in this file do?
```

The response streams into the cell's output as Markdown.

## Magics reference

- **`%ask <prompt>`** — line magic for a one-line prompt.
  - Prefix with `model|` or `provider:model|` to override the configured
    default model for just this call, e.g. `%ask openrouter:openai/gpt-4o|summarize this`.
  - Add `--debug` anywhere on the line to also show the full request/response
    payload.
- **`%%ask [model]`** — cell magic; the whole cell body is the prompt
  (multi-line is fine, and it can reference images via Markdown
  `![]()`/`<img>` syntax or bare local file paths — they're inlined as
  base64). An optional model name on the magic line overrides the default
  for this call. Also supports `--debug`.
- **`%%skip`** — runs the cell normally, but the cell is left out of the
  context sent to the model entirely. Use it for scratch or exploratory
  cells you don't want the model to see.
- **`%%pin`** — runs the cell normally; its content and output are *always*
  included in context and are exempt from the auto-trim behavior below. Use
  it to protect a fact, constant, or definition you don't want dropped over
  a long session.
- **`%%agent [model]`** — a multi-step, tool-using agent instead of a
  one-shot answer. See below.

## `%%agent` — the agentic magic

Where `%%ask` answers once from what the notebook *shows*, `%%agent` can take
several steps and look at what the kernel actually *holds* — including
variables no cell output ever displayed. It runs
[deepagents](https://docs.langchain.com/oss/python/deepagents/) in-process, as
a backend parallel to `%%ask`'s; `%%ask` is unchanged.

It needs the optional extra:

```bash
pip install 'shidoshi[agent]'   # or: uv add 'shidoshi[agent]'
```

```
%%agent
Which of my dataframes has missing values, and where?
```

The answer renders as Markdown with the agent's steps in a collapsed
**🧠 Agent steps** panel above it.

### Tool rungs

`--tools` chooses what the agent is allowed to do. **The default is `actor`:
model-written code runs in your live namespace, with no confirmation step.**
That is the reason `%%agent` exists rather than being a slower `%%ask`, so it
does not sit behind a flag — but know that it is what a bare `%%agent` cell
does.

| rung | what it can do |
|---|---|
| `none` | answer from context alone |
| `observer` | list and inspect kernel variables |
| `proposer` | the above, plus propose a cell for you to run |
| `actor` *(default)* | the above, plus **execute code in your kernel directly** |

`--no-tools` is the way back down — short for `--tools none`, for when you want
the conversation without the hands. `--tools proposer` is the middle ground: the
agent hands you a cell and *your Run button is the approval step*, with no
separate permission prompt to click.

Set a different default with `agent_tools` in config; `agent_tools = "proposer"`
restores the gated behavior for every cell.

### Memory

Each notebook gets one conversation thread, seeded once from the cells above
the first `%%agent` call. Later cells continue that conversation rather than
rebuilding context each time.

The thread lasts as long as the kernel and no longer. This is deliberate: what
the agent remembers is largely *kernel state* — variables it listed, values it
read — and a restart destroys exactly that. A conversation that outlived the
kernel would keep tool results reporting variables that no longer exist, worded
as fact. So a restart clears the thread and the notebook is read again as it
currently stands, which is what rerunning it from the top means anyway.

Use `--fresh` to opt out of the thread entirely and get `%%ask`-style
behavior — context rebuilt from cells, nothing remembered — or `--thread NAME`
to keep a side conversation separate.

### Inspecting a turn

`--debug` works as it does for `%%ask`, adapted to a graph that runs more than
once. Above the answer you get the request panel — provider and model, the
composed system prompt, every message the model will see, and the tools this
rung binds — followed by a live panel of raw LangGraph events as the turn runs.
Messages already in the thread are labelled `from thread`, because on a
continued conversation only your new prompt is passed in; the rest comes from
the checkpoint.

The stream panel is LangGraph's fullest per-step view — `task`, `task_result`
and `checkpoint` events with step numbers, the middleware nodes that a plain
update stream never names, and a per-task error when one fails. Failed steps are
flagged in the summary line so you don't have to expand them to find the one
that broke. A running token count sits above it, for both providers.

Token deltas are collapsed into a per-node count, so what you read is the node
transitions — `model` → `tools` → `model` — rather than several hundred
one-token lines.

An unrecognised flag is an error rather than a no-op, so a typo'd `--tools`
tells you instead of quietly running with the default.

### Getting help

`%ask --help` and `%agent --help` print the flags, and for `%%agent` the rung
table. Use the **single-`%` line form**: IPython rejects a cell magic whose body
is empty before the magic itself runs, so `%%agent --help` on its own can never
reach us.

`%agent [model |] your request` is also a one-line shorthand for `%%agent`,
matching `%ask`. Flags are cell-form only — on one line a bare token cannot be
told apart from the first word of a question.

## How context is built

Every prior cell in the notebook — up to the one you're currently running,
and accounting for kernel restarts — is turned into conversation history
automatically:

- **Markdown cells** become background text/image context (treated as notes
  or reference material, not instructions).
- **Regular code cells** appear as fenced code plus their text/image
  outputs.
- **Prior `%ask` / `%%ask` cells** become real user/assistant turns. Their
  responses are reused from a per-cell cache rather than re-sent, so
  replaying history doesn't resend answers the model already produced.
- **`%%skip` cells** are dropped entirely.
- **`%%pin` cells** are always kept.

## Automatic context-length handling

If a request is rejected for exceeding the model's context window, shidoshi
automatically retries, dropping the oldest trimmable history units first
(markdown cells, then plain code cells, then whole ask+response pairs —
`%%pin` cells are never dropped), up to 20 times. A banner reports how many
cells were dropped so you know context shrank.

## Providers & tools

- **`openai`** (default) — uses the OpenAI Responses API.
- **`openrouter`** — uses OpenRouter's chat-completions API; select it with
  the `provider:model` prefix, e.g. `openrouter:anthropic/claude-3.5-sonnet`.

Every request currently has the built-in `web_search` tool attached, so the
model can search the web when it needs current information. (A `web_fetch`
tool also exists in the codebase but isn't wired into the magics yet — not
available today.)

## Debug mode

Add `--debug` to `%ask`/`%%ask` to render a collapsible, syntax-highlighted
panel showing exactly what was sent (system prompt, full message history,
tools) and every raw event streamed back — useful when the model's behavior
is surprising and you want to see the actual payload.

## Configuration

shidoshi reads TOML config, layered as defaults → `~/.shidoshi/config.toml`
→ `./.shidoshi/config.toml` (project config overrides user config):

```toml
default_model = "gpt-5.5"   # model used when none is specified
reasoning_effort = "low"    # OpenAI only
ask_color = "#eafbea"       # highlight color for %ask/%%ask cells
skip_color = "#ececec"      # highlight color for %%skip cells

agent_model = ""            # %%agent model; falls back to default_model
agent_tools = "actor"       # none | observer | proposer | actor
```

## Development

```bash
uv sync
uv run pytest tests/unit tests/btp -v
```

Integration tests under `tests/integration/` require a live `OPENAI_API_KEY`
(or a proxy via `OPENAI_BASE_URL`) and are run with:

```bash
uv run pytest tests/integration/ -v -m integration
```

## License

Apache License 2.0 — see [LICENSE](LICENSE).
