Metadata-Version: 2.4
Name: tokensaver-ai
Version: 1.3.0
Summary: Compile repositories into minimal agent context with exact token compression metrics.
Author: Rachit Gupta
License-Expression: MIT
Keywords: agents,codebase,benchmark,tokens,context,mcp
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Utilities
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7.0
Provides-Extra: mcp
Requires-Dist: fastmcp>=3.0.0; extra == "mcp"
Dynamic: license-file

# TokenSaver

**Stop making coding agents reread your whole repo.**

TokenSaver compiles your repository into a compact context bundle that agents can use before they touch raw source. Instead of burning tokens re-parsing files on every task, agents start with the structure that actually matters: modules, APIs, routes, config, commands, and exact compression metrics.

That means faster repo understanding, less wasted context, and more consistent answers across large codebases.

Used across 11 anonymized real-world repositories with up to 77.15x measured compression.

Works with **Cursor**, **Claude Code**, **Codex**, **Windsurf**, and **MCP-enabled workflows**.

---

## Who It’s For

TokenSaver is for teams using coding agents on repositories that are too large, too modular, or too repetitive to keep re-reading from scratch.

It helps most when you have:

- large monorepos or multi-module apps
- mobile + web + backend repositories
- repeated agent sessions in the same codebase
- architecture, API, navigation, or config questions
- agents that keep spending context on repo discovery before doing useful work

## When TokenSaver Helps Most

TokenSaver is most useful when the hard part is **understanding the repo**, not writing one isolated function.

Typical cases:

- onboarding to an unfamiliar codebase
- answering "where does this live?" or "how is this wired?"
- tracing APIs, routes, modules, and config
- keeping agent context stable across multiple tasks
- reducing repeated raw-source reads in long sessions

## Why Not Just Let the Agent Read the Repo?

You can. But on larger repositories, that gets expensive and noisy fast.

Without TokenSaver:

- agents repeatedly crawl raw source to rediscover structure
- context gets wasted on files that are not immediately relevant
- answers can vary across sessions depending on what was read first
- large codebases take longer to orient around before real work begins

With TokenSaver:

- agents start with a compact map of the repo
- architecture, API, route, and config questions are easier to answer from artifacts first
- raw source is read only when implementation detail is actually needed
- token usage becomes measurable instead of guesswork

## Why TokenSaver?

TokenSaver helps agents start with structure instead of raw source:

| Without TokenSaver | With TokenSaver |
|---|---|
| Agent reads 800K tokens of raw source | Agent reads ~15K tokens of structured context |
| Every prompt re-parses the entire codebase | Artifacts are pre-built, cached, and incremental |
| No awareness of project structure | Module graph, API index, route map, config keys |
| Context window fills up fast | Exact per-artifact and repo-level compression metrics |

**Real-world proof:** up to **77.15x** measured compression across 11 anonymized real-world repositories and 15/15 public fixtures passing with 100% framework detection.

**Published real-world benchmarks** (anonymized, exact `tiktoken` counts):

| Repo | Framework | Plugin | Files | Source Tokens | Bundle Tokens | Compression |
|---|---|---|---:|---:|---:|---:|
| Confidential Flutter App A | `flutter` | `flutter` | 10,264 | 8,510,170 | 157,867 | **53.91x** |
| Confidential Flutter App B | `flutter` | `flutter` | 105 | 222,337 | 2,882 | **77.15x** |
| Confidential React Native App A | `react_native` | `react_native` | 896 | 804,476 | 28,466 | **28.26x** |
| Confidential React Native App B | `react_native` | `react_native` | 570 | 71,748 | 5,063 | **14.17x** |
| Confidential Node Backend A | `node` | `generic` | 32 | 19,998 | 5,498 | **3.64x** |
| Confidential Next.js App A | `nextjs` | `nextjs` | 59 | 18,747 | 1,999 | **9.38x** |
| Confidential Android App A | `android_native` | `android_native` | 2,255 | 4,427,242 | 2,180,896 | **2.03x** |
| Confidential iOS App A | `ios_swift` | `ios_swift` | 2,449 | 5,903,230 | 84,544 | **69.82x** |
| Confidential React Web App A | `react` | `react_web` | 27 | 16,390 | 2,332 | **7.03x** |
| Confidential Angular App A | `angular` | `angular` | 456 | 254,702 | 23,379 | **10.89x** |
| Confidential Angular App B | `angular` | `angular` | 140 | 101,390 | 12,850 | **7.89x** |

These are exact measured results from 11 anonymized real-world repositories across 8 frameworks. Compression ratios vary by project structure and framework. Public fixture benchmarks are used for contract validation and support coverage, not headline compression claims. See [Published Results](benchmarks/results.md) for per-artifact breakdowns.

---

## Quick Start

### Standard install

```bash
python3 -m pip install --user tokensaver-ai
```

If `tokensaver` is not found after install, add your user Python bin directory to `PATH`:

```bash
export PATH="$(python3 -m site --user-base)/bin:$PATH"
```

### Install with MCP support

```bash
python3 -m pip install --user "tokensaver-ai[mcp]"
```

With the MCP extra installed, `tokensaver init .` also writes Cursor and Claude MCP configs automatically.

### Add to any repo

```bash
cd /path/to/your-repo
tokensaver init .
```

This one command:

- builds the TokenSaver bundle
- writes Cursor, Claude Code, Codex, and Windsurf rule files
- writes Cursor and Claude MCP configs when MCP support is installed
- prints the repo’s compression summary

## How Agents Use It

Instead of opening raw source first, agents can start from the generated bundle:

- `PROJECT_SUMMARY.json` for framework, entrypoints, and project shape
- `COMMANDS.json` for run, build, test, and lint commands
- `MODULE_GRAPH.json` for structure and dependencies
- `API_INDEX.json` for endpoints and services
- `ROUTE_INDEX.json` for navigation and pages
- `CONFIG_INDEX.json` for env/config keys
- `METRICS.json` for exact compression numbers

That means agents begin with structure first, then drill into source only when necessary.

## Before and After

Without TokenSaver:

- agent opens many files just to answer "how does this repo work?"

With TokenSaver:

- agent reads the bundle first
- finds the relevant module, route, API, or config quickly
- opens only the few source files needed for implementation detail

### Scan

```bash
tokensaver scan /path/to/repo
```

Outputs exact file counts, token counts, detected framework, and language breakdown.

### Build

```bash
tokensaver build /path/to/repo
```

Generates 7 artifacts in `docs/tokensaver/` and auto-installs agent integrations for **Cursor**, **Claude Code**, **Codex**, and **Windsurf**.

### Incremental Rebuild

```bash
tokensaver build /path/to/repo
# Only changed artifacts are regenerated (SHA-256 diffing)

tokensaver build /path/to/repo --force
# Force full rebuild
```

### Impact Analysis

```bash
tokensaver impact /path/to/repo
# Shows blast-radius: which modules, APIs, routes are affected by recent changes

tokensaver impact /path/to/repo --files src/auth/login.py,src/models/user.py
```

### MCP Server

```bash
python3 -m pip install --user "tokensaver-ai[mcp]"
cd /path/to/repo
tokensaver init .
tokensaver serve /path/to/repo
# Starts the TokenSaver MCP server for interactive querying
```

---

## Supported Frameworks (17 stacks, 12 plugins)

### First-Class Plugins (deep extraction)

| Framework | Plugin | Detection | What's Extracted |
|---|---|---|---|
| **Flutter** | `flutter` | `pubspec.yaml` | GetX routes, Dart API URLs, RemoteConfig keys, module graph |
| **React Native** | `react_native` | `package.json` → `react-native` | Stack.Screen navigation, Axios/fetch APIs, RN Config, module graph |
| **Next.js** | `nextjs` | `package.json` → `next` | App Router + Pages Router, API routes, server actions, `next.config`, `NEXT_PUBLIC_*` env |
| **Workspace (multi-app)** | `workspace` | multiple nested app roots such as `frontend/` + `backend/` | Merged artifacts from child apps, including nested routes, APIs, config, module graph, and commands |
| **Angular** | `angular` | `package.json` → `@angular/core` | RouterModule routes, HttpClient APIs, `environment.ts` config, `@Component`/`@Injectable` |
| **React (web)** | `react_web` | `package.json` → `react` | React Router v5+v6, fetch/axios APIs, `REACT_APP_*` env, lazy imports |
| **FastAPI** | `python_web` | deps → `fastapi` | Decorator routes, Pydantic models, middleware, env config |
| **Django** | `python_web` | `manage.py` or deps → `django` | URL patterns, ORM models, middleware, `settings.py` keys |
| **Flask** | `python_web` | deps → `flask` | Route decorators, models, middleware, env config |
| **Spring Boot** | `spring_boot` | `build.gradle`/`pom.xml` → `spring-boot` | `@GetMapping`/`@PostMapping`, `@Entity` models, JPA repos, `application.properties`/`.yml` |
| **Android Native** | `android_native` | `build.gradle` (non-Spring) | Activities, Fragments, Jetpack Compose `composable()` routes, Retrofit APIs, `BuildConfig`, `strings.xml` |
| **iOS (Swift)** | `ios_swift` | `.xcodeproj` / `Package.swift` | SwiftUI Views, NavigationLink, UIKit ViewControllers, URLSession/Alamofire APIs, UserDefaults, `@AppStorage`, Info.plist |
| **Go** | `go` | `go.mod` | `net/http`, Gin, Chi, Echo, Fiber routes, structs, `os.Getenv`, Viper config, `go.mod` deps |

### Generic Fallback (all other projects)

| Framework | Detection | What's Extracted |
|---|---|---|
| Node.js / Express | `package.json` | Express routes, mount paths, `process.env`, module graph |
| Python (generic) | `*.py` files | Decorator routes, env config, module graph |
| PHP | `composer.json` or `*.php` files | Generic PHP routes, env config, composer scripts, module graph |
| Rust | `Cargo.toml` | Module graph, framework detection |

---

## Benchmark Results (15 fixture suite)

All 15 fixtures pass with `ok` status, 100% framework detection accuracy:

| Fixture | Framework | Status | Plugin |
|---|---|---|---|
| Flutter | `flutter` | ok | `flutter` |
| React Native | `react_native` | ok | `react_native` |
| Next.js | `nextjs` | ok | `nextjs` |
| Angular | `angular` | ok | `angular` |
| React Web | `react` | ok | `react_web` |
| FastAPI | `fastapi` | ok | `python_web` |
| Django | `django` | ok | `python_web` |
| Spring Boot | `spring_boot` | ok | `spring_boot` |
| Android Native | `android_native` | ok | `android_native` |
| iOS Swift | `ios_swift` | ok | `ios_swift` |
| Go | `go` | ok | `go` |
| Node.js | `node` | ok | `generic` |
| Python | `python` | ok | `generic` |
| PHP | `php` | ok | `generic` |
| Workspace | `workspace` | ok | `workspace` |

---

## Build Outputs

`tokensaver build` generates these artifacts:

| Artifact | Contents |
|---|---|
| `PROJECT_SUMMARY.json` | Framework, languages, entrypoints, manifests |
| `COMMANDS.json` | Build/dev/test/lint commands from package.json, Makefile, CI |
| `MODULE_GRAPH.json` | Module names, file counts, token counts |
| `API_INDEX.json` | API endpoints with methods, paths, source locations |
| `ROUTE_INDEX.json` | UI routes / URL patterns with navigation graph |
| `CONFIG_INDEX.json` | Environment variables, settings keys, config references |
| `METRICS.json` | Exact compression metrics per artifact and overall |

Every artifact carries `schema_version` in `_meta`. See [Output Schema](docs/OUTPUT_SCHEMA.md).

---

## Agent Integrations

`tokensaver build` automatically installs integration files so agents pick up context without manual referencing:

| Agent | File Generated | How It Works |
|---|---|---|
| **Cursor** | `.cursor/rules/tokensaver.mdc` | Auto-injected as context rule |
| **Claude Code** | `CLAUDE.md` | Read by Claude on project open |
| **Codex** | `AGENTS.md` | Read by Codex on project open |
| **Windsurf** | `.windsurfrules` | Read by Windsurf on project open |
| **Cursor MCP** | `.cursor/mcp.json` | Interactive MCP querying |
| **Claude MCP** | `.mcp.json` | Interactive MCP querying |

---

## Architecture

```
tokensaver/
  core/             # Scan, build orchestration, plugin protocol, shared models
    plugin_api.py    # TokenSaverPlugin protocol
    registry.py      # Plugin registry (ordered matching)
    helpers.py       # Shared regex patterns, utilities
    models.py        # ArtifactResult, BuildContext
  plugins/           # Framework-specific extractors
    flutter.py       # Flutter (GetX, Dart APIs, RemoteConfig)
    react_native.py  # React Native (Stack navigation, Axios)
    nextjs.py        # Next.js (App Router, API routes, server actions)
    angular.py       # Angular (RouterModule, HttpClient, environment.ts)
    react_web.py     # React web (React Router, fetch/axios, REACT_APP_*)
    python_web.py    # FastAPI / Django / Flask
    spring_boot.py   # Spring Boot (annotations, JPA, properties)
    android_native.py # Android (Compose, Retrofit, BuildConfig, strings.xml)
    ios_swift.py     # iOS (SwiftUI, UIKit, URLSession, UserDefaults)
    go_mod.py        # Go (net/http, Gin, Chi, Echo, Fiber)
    generic.py       # Fallback for all other stacks
  scanner.py         # Framework detection + token accounting
  build.py           # Build orchestration + incremental diffing
  snapshot.py        # SHA-256 snapshot for incremental builds
  impact.py          # Blast-radius change-impact analysis
  integrations.py    # IDE/agent config file generation
  mcp_server.py      # MCP server (FastMCP)
  benchmark.py       # Reproducible benchmarking + suite runner
tokensaver_cli.py    # CLI entry point
```

---

## CLI Reference

```bash
tokensaver scan <path>                                    # Scan and report token counts
tokensaver init <path> [--output-dir <dir>] [--force]     # One-shot repo onboarding
tokensaver build <path> [--output-dir <dir>] [--force]    # Build artifacts
tokensaver impact <path> [--files f1,f2]                  # Blast-radius analysis
tokensaver serve <path>                                   # Start MCP server
tokensaver metrics <path>                                 # Print existing metrics
tokensaver benchmark <path>                               # Run reproducible benchmark
tokensaver benchmark-suite <manifest>                     # Run multi-repo benchmark suite
tokensaver diff-snapshots <old> <new>                     # Compare benchmark snapshots
```

---

## Adding a New Plugin

TokenSaver's plugin system is designed for easy extension. Each plugin is ~200-300 lines:

1. Create `tokensaver/plugins/your_framework.py`
2. Implement the `TokenSaverPlugin` protocol: `name`, `frameworks`, `build_artifacts(ctx)`
3. Return 4 artifacts: `module_graph`, `api_index`, `route_index`, `config_index`
4. Register in `tokensaver/core/registry.py`
5. Add framework detection in `tokensaver/scanner.py`

```python
@dataclass(frozen=True)
class YourPlugin:
    name: str = "your_framework"
    frameworks: set[str] = frozenset({"your_framework"})

    def build_artifacts(self, ctx: BuildContext) -> list[ArtifactResult]:
        return [
            build_module_graph(ctx),
            build_api_index(ctx),
            build_route_index(ctx),
            build_config_index(ctx),
        ]
```

---

## Metric Semantics

TokenSaver reports only **measurable** compression metrics — no estimates, no dollar claims:

| Metric | Definition |
|---|---|
| `source_tokens` | Exact `tiktoken` (o200k_base) count for source files |
| `output_tokens` | Exact count for generated artifact |
| `compression_ratio` | `source_tokens / output_tokens` |
| `union_source_tokens` | Deduplicated token count across all source files |
| `bundle_tokens` | Total tokens across all artifacts |

---

## Benchmarking

```bash
tokensaver benchmark .                    # Single repo
tokensaver benchmark-suite manifest.json  # Multi-repo suite
tokensaver benchmark-suite manifest.json --public-only  # Safe for publishing
tokensaver diff-snapshots old.json new.json  # Regression detection
```

See [Benchmark Guide](benchmarks/README.md) for manifest format.

---

## Documentation

- [Output Schema](docs/OUTPUT_SCHEMA.md) — canonical artifact shapes
- [Compatibility Policy](docs/COMPATIBILITY.md) — versioning and backward compatibility
- [Known Limitations](docs/KNOWN_LIMITATIONS.md) — current boundaries
- [Benchmark Guide](benchmarks/README.md) — manifest format, privacy workflow
- [Contributing](CONTRIBUTING.md)
- [Changelog](CHANGELOG.md)

---

## License

MIT — commercial use allowed.
