Metadata-Version: 2.4
Name: yak-browser-use
Version: 0.8.4.dev0
Summary: A clean, learnable browser automation framework — Chat mode + Preset replay via Playwright CDP
Author-email: SlimeSB <x56277534@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/SlimeSB/yak-browser-use
Project-URL: Source, https://github.com/SlimeSB/yak-browser-use
Project-URL: BugTracker, https://github.com/SlimeSB/yak-browser-use/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: ddddocr>=1.6.1
Requires-Dist: openai>=1.0.0
Requires-Dist: openpyxl>=3.1.0
Requires-Dist: playwright>=1.48.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn[standard]>=0.30.0
Requires-Dist: websockets>=15.0

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/logo.png">
    <img src="assets/logo.png" alt="yak-browser-use logo" width="240">
  </picture>
</p>

<h1 align="center">Yak Browser-Use</h1>

<p align="center">
  <strong>CHAT · BROWSER · AUTOMATE</strong>
</p>

<p align="center">
  <em>An AI Agent product that chats with you while operating the browser</em>
</p>

<p align="center">
  <img src="https://img.shields.io/pypi/v/yak-browser-use?style=flat-square&logo=pypi&label=PyPI" alt="PyPI">
  <img src="https://img.shields.io/pypi/dm/yak-browser-use?style=flat-square&label=downloads" alt="Downloads">
  <img src="https://img.shields.io/github/stars/SlimeSB/yak-browser-use?style=flat-square&logo=github" alt="GitHub Stars">
  <img src="https://img.shields.io/github/actions/workflow/status/SlimeSB/yak-browser-use/ci.yml?branch=main&style=flat-square&label=CI" alt="CI">
  <img src="https://img.shields.io/badge/python-%E2%89%A53.12%20|%203.13-blue?style=flat-square&logo=python" alt="Python ≥3.12 / 3.13">
  <img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="MIT License">
  <img src="https://img.shields.io/badge/status-beta-orange?style=flat-square" alt="Alpha">
  <img src="https://img.shields.io/badge/Playwright-ready-45ba4b?style=flat-square&logo=playwright" alt="Playwright">
  <img src="https://img.shields.io/badge/Electron-desktop-47848F?style=flat-square&logo=electron" alt="Electron Desktop">
  <img src="https://img.shields.io/badge/Web%20UI-uvx-8A2BE2?style=flat-square&logo=web" alt="Web UI">
  <a href="./README.zh-CN.md"><img src="https://img.shields.io/badge/README-中文-blue?style=flat-square" alt="中文"></a>
</p>
<p align="center">
  <a href="./README.md">English</a> · <a href="./README.zh-CN.md">简体中文</a>
</p>

<p align="center">
  <strong>UI Screenshots</strong>
</p>

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/UI-chat-en.png">
    <img src="assets/UI-chat-en.png" alt="Chat Mode" width="45%">
  </picture>
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/UI-preset-en.png">
    <img src="assets/UI-preset-en.png" alt="Preset Mode" width="45%">
  </picture>
  <br/>
  <sub>Chat Mode · Preset Mode</sub>
</p>

---

## What is Ybu?

**yak-browser-use** (aliased as **ybu**) is a browser automation AI Agent product. Its core interaction model:

> **You chat with the Agent → Agent controls the browser → You watch it happen in real-time**

Two modes:
- **Chat Mode** — natural language conversational control, Agent browses while chatting
- **Preset Mode** — replay recorded pipelines, execute pre-defined steps automatically (zero LLM token consumption)

Built on [Playwright](https://playwright.dev/) `connect_over_cdp()` and an OpenAI-compatible LLM client.

> **Purpose-built architecture.** ybu is an independent codebase with its own conversation loop, progressive snapshot engine, CDP integration, and pipeline compiler — designed specifically for this project. It has zero browser-use dependencies and shares no code with any other browser automation framework.

---

## Features

### Core

| Feature | Description |
|---------|-------------|
| **Chat Mode** | Conversational browser control — tell the Agent what to do in natural language |
| **Preset Mode** | Execute pre-defined YAML pipelines with DAG step dependencies |
| **Three Step Types** | `browser_ops` (direct browser actions), `tool_name` (tool execution), `goal_description` (LLM-driven goals) |
| **Pipeline Compiler** | YAML-based pipeline definition with validation, step type inference, and chat-to-pipeline compilation |
| **Auto Recovery** | When a preset pipeline step fails, the Agent enters a recovery loop to diagnose and fix |
| **Iteration Budget** | Configurable LLM round-trip budget (default 50) with CDP reconnect pause/resume |

### Browser Control

| Feature | Description |
|---------|-------------|
| **CDP Connection** | Connects to Chrome via Playwright `connect_over_cdp()` — no browser extension needed |
| **User / Isolated Mode** | Connect to your existing Chrome, or launch an isolated Chrome instance |
| **Rich Browser Ops** | `goto`, `click`, `fill`, `snapshot`, `scroll`, `lookup_selector`, `hover`, `unhover`, `focus`, `select`, `clear`, `keyboard`, `press_key`, `type_text`, `navigate`, `wait`, `tab`, `copy`, `paste` |
| **Four Snapshot Modes** | `aria` (YAML semantic tree, LLM-friendly), `a11y` (structured element list), `progressive` (DOM depth scan with density-adaptive folding), `full` (screenshot+HTML) |
| **Element Highlighting** | Visual element badges on the page — a11y/progressive/off modes |
| **Multi-tab Support** | Automatic tab switching, new tab handling, per-page highlights |
| **Download Management** | CDP-based download routing to run-specific directories with Playwright fallback |
| **Health Check** | Periodic CDP heartbeat to detect browser disconnection |
| **Process Watcher** | Monitors spawned browser subprocess for auto-disconnect on window close/crash |

### Pipeline System

| Feature | Description |
|---------|-------------|
| **YAML Pipeline** | Declarative pipeline definition with name, description, steps, constants, URL aliases |
| **DAG Step Resolution** | Steps declare `depends_on` — the step machine resolves and executes in order |
| **Step State Machine** | PENDING → RUNNING → SUCCESS/FAILED/SKIPPED/PENDING_REVIEW with retryable error classification |
| **Verification Checks** | Post-step verification: `url_contains`, `element_exists`, `text_contains`, `element_visible`, `output_exists`, `file_contains`, `js_expression`, `json_field_exists` |
| **Pipeline Tools** | Agent can `create`, `view`, `update_step`, `add_step`, `remove_step`, `compile` pipelines during execution |
| **Chat-to-Pipeline** | Compile a chat session's tool calls into a reusable pipeline.yaml |
| **Version Snapshots** | Pipeline versions tracked in workspace for rollback |

### Tool System

| Feature | Description |
|---------|-------------|
| **Rich Tool Registry** | 40+ tools: browser control, data extraction, pipeline management, file I/O, skill system |
| **Data Tools** | `extract_fields`, `extract`, `read_data` (progressive offset/limit + format conversion), `file_read`, `file_write`, `format_convert` |
| **Shared Store** | Runtime data bus — tools pass data between steps via `{*key}` (whole-value pointer), `${key}` (inline template interpolation), and `{key}` (deprecated compat) syntax |
| **Sensitive Data Masking** | Automatic masking of credentials (API keys, passwords, tokens) in logs and tool results |
| **Todo System** | Built-in todo management for Agent task tracking |
| **Edit Pipeline** | Agent can edit pipeline.yaml content during execution |
| **Captcha Handling** | Captcha detection and processing tool |

### UI & API

| Feature | Description |
|---------|-------------|
| **Web UI** | Browser-based UI — launch with `uvx yak-browser-use web`, zero setup |
| **Electron Desktop** | Cross-platform desktop app with native Chrome DevTools Protocol integration |
| **REST API** | FastAPI-based server with endpoints for chat, run, status, Chrome management, provider config |
| **WebSocket Events** | Real-time event streaming for pipeline progress, LLM streaming, logs |
| **Provider Config** | LLM provider configuration UI with test endpoint and preset models |
| **Session Persistence** | Chat sessions saved to disk per pipeline, with index management and archive |
| **Unified Logs** | Combined backend/electron/LLM log viewer with follow mode and source filtering |

### Workspace

| Feature | Description |
|---------|-------------|
| **Per-pipeline Workspace** | Isolated directories for runs, sessions, versions, tools, downloads |
| **Run Lifecycle** | Run status tracking (pending/running/completed/failed/crashed/cancelled) with metadata |
| **Crash Detection** | Automatic detection and marking of crashed runs on restart |
| **Run Cleanup** | Configurable max runs retention with oldest-first cleanup |
| **Path Guard** | Security guard against path traversal in workspace operations |

### Execution Engine

| Feature | Description |
|---------|-------------|
| **Three Executors** | Browser op executor, tool executor, goal executor — each with core + pipeline wrappers |
| **Step Machine** | DAG-based step lifecycle with execution tree export |
| **Error Classification** | Retryable vs non-retryable error classification with compensation registry |
| **Compensation System** | Error compensation handlers for step failure recovery |
| **Streaming LLM** | Streaming LLM call with text/reasoning/tool deltas for real-time UI updates |
| **Event System** | Structured event sink for pipeline progress, errors, and state changes |

---

## Quick Start

### One-command (no install required)

```bash
uvx yak-browser-use web
```

Opens the Web UI in your browser — zero setup. The first run auto-installs the package and dependencies.

### Prerequisites

| Dependency | Version | Install |
|------------|---------|---------|
| Python | ≥ 3.12 | [python.org](https://python.org) |
| [uv](https://docs.astral.sh/uv/) | ≥ 0.4 | `powershell -c "irm https://astral.sh/uv/install.ps1 \| iex"` |
| Node.js | ≥ 18 | [nodejs.org](https://nodejs.org) |
| Chrome / Chromium | ≥ 120 | Your existing Chrome, or `uv run playwright install chromium` |

> `uvx` launches directly from PyPI — no local Python/Node.js setup needed.

### Install

```bash
# Windows one-click
install.bat

# Or manual three steps
cd backend
uv sync                              # Install Python deps
uv run playwright install chromium   # Install Playwright Chromium
cd ../electron
npm install                          # Install Electron frontend deps
```

### Start

```bash
# Quickest — launch Web UI from PyPI (no local setup needed)
uvx yak-browser-use web

# Or after local install:
cd backend
uv run python -m yak_browser_use web      # Web UI (browser-based)
uv run python -m yak_browser_use serve     # REST API server
uv run python -m yak_browser_use --help    # All CLI commands

# Electron desktop (requires Node.js)
cd electron
npm run electron:dev
```

### Configure Provider

Create `userdata/provider.json` (or configure via Electron Settings → LLM Provider):

```json
{
  "model": "deepseek-chat",
  "api_key": "sk-xxx...xxxx",
  "api_base": "https://api.deepseek.com"
}
```

---

## Commands

```text
ybu run <path>                 Execute a pipeline.yaml
ybu serve [--port PORT]        Start the REST API server
ybu web                        Start the Web UI (browser, no Electron)
ybu logs [-f] [--source all]   View unified logs
```

> CLI commands: `serve`, `run`, `web`, `logs`. Config via Web UI / Electron Settings (not CLI subcommands).

---

## How It Works

### Two-Layer Architecture

```
┌─────────────────────────────────────────────────────┐
│                  Orchestration Layer                  │
│  ┌──────────┐  ┌──────────┐  ┌───────────────────┐  │
│  │ Chat Mode │  │ Preset   │  │ Pipeline Compiler  │  │
│  │ (Agent)   │  │ (Runner) │  │ (YAML → Steps)    │  │
│  └─────┬─────┘  └────┬─────┘  └────────┬──────────┘  │
│        │              │                 │             │
│  ┌─────┴──────────────┴─────────────────┴──────────┐  │
│  │           Step Machine (DAG Engine)              │  │
│  │  PENDING → RUNNING → SUCCESS/FAILED/SKIPPED     │  │
│  └─────────────────────┬───────────────────────────┘  │
└────────────────────────┼──────────────────────────────┘
                         │
┌────────────────────────┼──────────────────────────────┐
│              Execution Layer                           │
│  ┌──────────┐  ┌──────────┐  ┌───────────────────┐   │
│  │ Browser  │  │ Tool     │  │ Goal Executor     │   │
│  │ Executor │  │ Executor │  │ (LLM-driven)      │   │
│  └────┬─────┘  └────┬─────┘  └────────┬──────────┘   │
│       │              │                 │              │
│  ┌────┴──────────────┴─────────────────┴──────────┐   │
│  │         Playwright Bridge (CDP)                 │   │
│  │  connect_over_cdp() → Chrome DevTools Protocol  │   │
│  └─────────────────────┬───────────────────────────┘   │
│                        │                               │
│  ┌─────────────────────┴───────────────────────────┐   │
│  │              Chrome Browser                       │   │
│  └─────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────┘
```

### Two Execution Modes

#### Chat Mode (Interactive)

```
User: "Open Baidu and search for coffee"
  → POST /api/chat { message: "..." }
    → StreamingLLMCall (streaming text + tool calls)
      → Tool Executor
        → browser_goto("https://www.baidu.com")
        → browser_fill("#kw", "coffee")
        → browser_click("#su")
        → browser_snapshot(mode="aria")
    → Response streamed via WebSocket
  → User sees browser update in real-time
```

**Key Points:**
- Agent has access to 40+ tools: browser control, data extraction, pipeline management, file I/O, skill system
- Each tool call triggers a CDP command via Playwright Bridge
- Page snapshots use ARIA semantic tree (token-efficient, LLM-friendly)
- Element highlights update after every action
- Session is persisted to disk for history

#### Preset Mode (Pipeline Replay)

```
POST /api/run { pipeline: "..." }
  └→ run_pipeline()
       ├→ Load and validate pipeline.yaml
       ├→ Create workspace run directory
       ├→ Step Machine resolves DAG dependencies
       ├→ For each step:
       │    ├→ browser_ops → execute_browser_step()
       │    ├→ tool_name   → execute_tool_step()
       │    └→ goal        → execute_goal_step() (LLM-driven)
       ├→ Post-step verification check
       ├→ On failure:
       │    ├→ Error classification (retryable vs non-retryable)
       │    ├→ Compensation handler (if registered)
       │    └→ Recovery loop: Agent diagnoses and fixes
       └→ Finalize run (execution tree, version snapshot)
```

**Key Points:**
- Steps declare `depends_on` for DAG execution order
- Three mutually exclusive step types: `browser_ops`, `tool_name`, `goal_description`
- Verification checks validate step success
- Auto-recovery: Agent re-enters with failure context, can edit pipeline and retry
- Execution tree written to run directory for post-mortem analysis

---

## Project Structure

```
yak-browser-use/
├── backend/
│   └── src/yak_browser_use/
│       ├── __main__.py          # CLI entry point (web/serve/run/logs)
│       ├── api/                 # FastAPI REST server + WebSocket
│       │   ├── routes.py        # All API endpoints
│       │   ├── service.py       # Chat service orchestration
│       │   ├── session_manager.py # Session CRUD + persistence
│       │   └── state.py         # Global engine state
│       ├── cdp/                 # Chrome DevTools Protocol layer
│       │   ├── playwright_bridge.py # Playwright CDP bridge
│       │   ├── launcher.py      # Isolated Chrome launcher
│       │   ├── discover.py      # CDP WebSocket URL discovery
│       │   ├── helpers.py       # CDP helper utilities
│       │   └── protocols.py     # BrowserBridge protocol
│       ├── cli/                 # CLI command implementations
│       ├── compiler/            # Pipeline YAML compiler
│       │   ├── schema.py        # Pydantic models (PipelineYaml, StepYaml)
│       │   ├── models.py        # Internal dataclasses (PipelineDef, StepDef)
│       │   ├── parser.py        # YAML parsing + param injection
│       │   ├── pipeline_store.py # Pipeline CRUD + YAML serialization
│       │   └── step_type.py     # Step type inference
│       ├── engine/              # Execution engine
│       │   ├── agent.py         # Streaming LLM call factory
│       │   ├── executor.py      # Browser/tool/goal executors
│       │   ├── runner_preset.py # Preset pipeline runner
│       │   ├── step_machine.py  # DAG step state machine
│       │   ├── ops.py           # Browser operation definitions
│       │   ├── state.py         # RunContext
│       │   ├── events.py        # Event sink
│       │   ├── delivery.py      # Event delivery
│       │   ├── _harness/        # Chat harness
│       │   │   ├── conversation_loop.py # Main chat loop
│       │   │   ├── tool_executor.py     # Tool call dispatcher
│       │   │   ├── tools.py            # Tool definitions
│       │   │   ├── pipeline_tools.py   # Pipeline management tools
│       │   │   ├── iteration_budget.py # LLM round-trip budget
│       │   │   ├── retry_utils.py      # Retry utilities
│       │   │   ├── error_classifier.py # Error classification
│       │   │   ├── tool_guardrails.py  # Tool usage guardrails
│       │   │   ├── skill_tools.py      # Skill-based tools
│       │   │   └── turn_context.py     # Turn context
│       │   └── _lifecycle/
│       │       └── compensation.py # Error compensation registry
│       ├── llm/                 # LLM client
│       ├── params/              # Parameter resolution
│       ├── prompts/             # System prompts
│       │   ├── chat/            # Chat mode prompts
│       │   ├── guidance/        # Agent guidance prompts
│       │   ├── guardrails/      # Guardrail prompts
│       │   └── skill/           # Skill prompts
│       ├── tools/               # Tool implementations
│       │   ├── registry.py      # Tool registry + schemas
│       │   ├── extract.py       # Data extraction
│       │   ├── file_read/write  # File I/O tools
│       │   ├── edit_pipeline.py # Pipeline editing
│       │   ├── todo.py          # Todo management
│       │   ├── captcha.py       # Captcha handling
│       │   └── ...
│       ├── utils/               # Utilities
│       └── workspace/           # Workspace management
│           ├── manager.py       # WorkspaceManager (runs, versions)
│           ├── session_store.py # Session persistence
│           └── path_guard.py    # Path traversal guard
├── electron/                    # Electron desktop app
│   └── src/
│       ├── main/                # Electron main process
│       ├── renderer/            # React UI
│       │   ├── components/      # UI components
│       │   ├── stores/          # State stores
│       │   ├── ws/              # WebSocket client
│       │   └── apiClient.ts     # REST API client
│       └── utils/               # Electron utilities
├── docs/                        # Architecture docs
├── userdata/                    # User data (provider config, etc.)
└── debug-page/                  # Debug page dumps
```

---

## Pipeline YAML Example

```yaml
name: baidu_search
description: Search Baidu and extract results
steps:
  - name: open_baidu
    description: Navigate to Baidu homepage
    browser_ops:
      - goto: "https://www.baidu.com"
    check:
      url_contains: baidu.com

  - name: search
    description: Search for a keyword
    depends_on: [open_baidu]
    browser_ops:
      - fill: { selector: "#kw", value: "yak-browser-use" }
      - click: "#su"
    check:
      element_exists: "#content_left"

  - name: extract_results
    description: Extract search result titles
    depends_on: [search]
    tool_name: extract_fields
    params:
      pattern: "search result titles"
    check:
      output_exists: true
```

---

## Shared Store — Data Bus Between Steps

The **shared store** is a runtime `dict` that acts as the data bus between pipeline steps and tool calls. Tools read from and write to it, enabling data flow without cluttering the LLM context window.

### Syntax Reference

| Syntax | Example | Behavior |
|--------|---------|----------|
| `{*key}` | `{*step_3_output}` | Whole-string pointer — replaces the entire parameter with the store value **as-is** (preserves type: list, dict, int, etc.) |
| `${key}` | `https://${host}/api` | Template interpolation — replaces `${key}` inline within a string. Result is **always a string**. Full-match behaves like `{*key}` |
| `{key}` | `{step_3_output}` | Deprecated compat — same as `{*key}` but without the `*` sigil. Prefer `{*key}` for new code |

### How It Works

```
Step 1: extract_fields → writes result to shared_store["extracted"]
Step 2: format_convert(source="{*extracted}", convert_to="csv")
         → {*extracted} resolves to the full list from Step 1
Step 3: file_write(content="Results: ${row_count} rows")
         → ${row_count} interpolates inline
```

Tools that write to shared_store accept an `output_to` parameter (e.g. `browser_source(output_to="page_html")`). Tools that read use the pointer syntax above. Use `data_keys` to list all keys, `data_browse(key=...)` to paginate through values, and `data_preview(key=...)` for a lightweight peek.

---

## Key Design Decisions

1. **PlaywrightBridge Unified Driver** — All browser operations go through `PlaywrightBridge` (`connect_over_cdp()`), gaining auto-wait / auto-scroll / auto-retry, plus health check heartbeat, process watcher, disconnect handling, and download management. `BrowserBridge` protocol (`cdp/protocols.py`) defines the interface contract.

2. **File as Contract** — `pipeline.yaml` is a static contract, strictly validated at compile time (step type mutual exclusion, check field validation, DAG dependency resolution), minimizing surprises at runtime.

3. **No sub-agent architecture** — The main LLM handles all tasks directly via `browser_*`, `todo`, and pipeline management tools — no sub-agent spawning, scheduling, or context management overhead. Complex steps are decomposed into subtasks the main LLM executes itself, keeping context in one place.

---

## Development

```bash
# Create and activate venv
cd backend
uv venv
source .venv/bin/activate   # Linux/macOS
.venv\Scripts\activate      # Windows

# Install dev dependencies
uv sync --dev

# Run tests
uv run pytest

# Coverage
uv run pytest --cov=.

# Open Chrome remote debugging port
chrome.exe --remote-debugging-port=9222
```

### Dev Commands

| Command | Description |
|---------|-------------|
| `uv run python -m yak_browser_use serve --port 8080` | Start API server |
| `uv run python -m yak_browser_use web` | Start Web UI (browser) |
| `uv run python -m yak_browser_use run path/to/pipeline.yaml` | Run a pipeline |
| `uv run python -m yak_browser_use logs -f` | Tail logs live |
| `uv run python -m yak_browser_use --help` | Show all CLI commands |
| `cd electron && npm run electron:dev` | Start Electron frontend |
| `cd electron && npm run dev:web` | Start Web frontend dev server (Vite HMR + proxy) |

---

## Architecture Docs

For a full architectural deep-dive (data flow diagrams, design principles, execution paths), see [`docs/architecture-overview.md`](docs/architecture-overview.md).

---

## License

MIT © 2026 Yak Browser-Use Contributors

See [`ACKNOWLEDGMENTS.md`](ACKNOWLEDGMENTS.md) for project references and contributor credits.

---

<p align="center">
  <img src="assets/logo.png" alt="yak" width="64">
  <br/>
  <sub>Built with yak power · Chat · Browser · Automate</sub>
</p>
