Metadata-Version: 2.4
Name: spoolcache
Version: 0.3.0
Summary: Patch-free, bounded-memory, rank-local NVMe KV cache for vLLM
Author: MiaAI Lab
License: MIT
Project-URL: Homepage, https://github.com/xudongcc/spoolcache
Project-URL: Repository, https://github.com/xudongcc/spoolcache.git
Project-URL: Documentation, https://github.com/xudongcc/spoolcache/blob/main/docs/README.md
Project-URL: Issues, https://github.com/xudongcc/spoolcache/issues
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Dynamic: license-file

# SpoolCache

SpoolCache persists vLLM KV cache shards on local NVMe so requests with a shared
prefix can reuse cached state across engine restarts. It runs as an external
vLLM connector, with fixed staging buffers and no changes to vLLM's source.

- Exact text and multimodal prefix identity, derived from the serving runtime.
- Complete cache-group and PP/TP participant agreement before a persistent hit.
- Runtime-aligned chained token files, buffered I/O and authenticated durable publication.
- Bounded inventory, capacity reclamation, background integrity checks and metrics.

SpoolCache is experimental. Runtime support is determined by public-interface
checks and discovered cache semantics. See [compatibility and evidence](docs/COMPATIBILITY.md)
for tested environments and limitations.

Published packages are available on [PyPI](https://pypi.org/project/spoolcache/).
See [release notes](https://github.com/xudongcc/spoolcache/releases) and the
[migration guide](docs/MIGRATION.md) when upgrading. For an older installation,
use the documentation from its release tag.

## Install

Install SpoolCache in the same Python environment as vLLM:

```bash
python -m pip install --upgrade spoolcache
spoolcache --help
```

Python 3.11+ is required. Serving additionally requires 64-bit Linux OFD file
locks and a working CUDA/vLLM runtime. The sole storage backend uses ordinary
buffered I/O, with no O_DIRECT or io_uring extension. The token-file backend
replaces the snapshot format used in 0.2.0; see the
[migration guide](docs/MIGRATION.md).
The package does not install vLLM, PyTorch, CUDA or model weights.
Production images use a verified wheel; see [deployment](docs/DEPLOYMENT.md).

## Use

For example, start `google/gemma-4-E2B-it` with SpoolCache:

```bash
vllm serve google/gemma-4-E2B-it \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --kv-transfer-config "$(spoolcache config)"
```

Keep your model's normal memory, topology and serving options. The configuration
renderer does not start vLLM; pass its output with `--kv-transfer-config`.
The startup gate rejects incompatible runtime and allocator contracts, including
an enabled `expandable_segments` allocator setting.

Ordinary requests need no extra SpoolCache field. Reuse requires an exact shared
prefix, compatible identities and a runtime-safe persistence boundary. A shorter
latency or nonzero API cached-token count alone does not prove an NVMe hit;
follow the [verification procedure](docs/DEVELOPMENT.md#verify-a-persistent-hit).

Per-request flags `spoolcache.skip_read` and `spoolcache.skip_write` independently
control persistent reads and writes through `kv_transfer_params`. Both default
to false and leave vLLM's own caches enabled. See [request controls](docs/CONFIGURATION.md#request-level-read-and-write-controls).

## Configure

The default connector uses the [token-file backend](docs/TOKEN_FILES.md).
Earlier snapshot/slot cache formats remain on disk but are not read or migrated.

| Variable | Default | Purpose |
| --- | --- | --- |
| `SPOOLCACHE_PATH` | `~/.cache/spoolcache` | Cache directory visible to the process; `~` is expanded. |
| `SPOOLCACHE_MAX_SIZE` | `200` | Capacity target per rank in GB (1 GB = 1024³ bytes); decimals allowed. |

Connector JSON uses `spoolcache_path` and `spoolcache_max_size`. Configuring the
connector enables both persistent reads and writes. To disable SpoolCache, omit its connector configuration. See the
[configuration reference](docs/CONFIGURATION.md).

Docker launchers also use `SPOOLCACHE_PATH` for the host directory, defaulting to
each host user's `~/.cache/spoolcache`. They map it to the configured container
path. See [container paths](docs/DEPLOYMENT.md#container-paths).

## Operate

The worker runs bounded integrity checks and capacity reclamation. Metrics
appear on vLLM's `/metrics` endpoint; deployment owns process restarts and
complete-group recovery. The CLI exposes `spoolcache config`; the old snapshot
`request` and `status` commands are removed.
Read the [operations guide](docs/OPERATIONS.md) before manipulating cache state.

## Develop

From this checkout:

```bash
uv sync --python 3.12 --locked --group dev
uv run --locked pytest -q
```

CPU storage tests use the real buffered token-file path and Linux file locks;
a GPU is unnecessary. GPU/runtime tests require their target image.
The [development guide](docs/DEVELOPMENT.md) covers wheel builds, containers,
multimodal checks and the isolated two-node harness.

## Documentation

[Documentation index](docs/README.md) · [Installation](docs/INSTALLATION.md) ·
[Configuration](docs/CONFIGURATION.md) · [Deployment](docs/DEPLOYMENT.md) ·
[Operations](docs/OPERATIONS.md) · [Development](docs/DEVELOPMENT.md) ·
[Design](docs/SPOOLCACHE_DESIGN.md) · [Releases](docs/RELEASE.md) ·
[Goals](docs/TODO_GOALS.md)

SpoolCache is distributed under the [MIT license](LICENSE).
