Metadata-Version: 2.5
Name: layerscale
Version: 0.5.1
Summary: Python client for the LayerScale inference server
Project-URL: Homepage, https://layerscale.ai
Author: LayerScale
License: MIT
Keywords: inference,layerscale,llm,sessions,streaming
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: anyio<5,>=4.0
Requires-Dist: httpx<1,>=0.28
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: typing-extensions<5,>=4.15
Requires-Dist: websockets<17,>=15
Provides-Extra: dev
Requires-Dist: pyright>=1.1.405; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: python-dotenv>=1.0; extra == 'dev'
Requires-Dist: respx>=0.22; extra == 'dev'
Requires-Dist: ruff>=0.15; extra == 'dev'
Description-Content-Type: text/markdown

# layerscale

Python client for the LayerScale inference server. Sync (`LayerScale`) and
async (`AsyncLayerScale`) clients with identical surfaces, typed requests
(TypedDicts) and responses (pydantic v2), both server SSE dialects, and a
WebSocket session socket.

Requires Python 3.9+. Runtime dependencies: `httpx`, `pydantic` v2,
`typing-extensions`, `anyio`, `websockets` (imported lazily, only if you open
a WebSocket).

```bash
pip install layerscale
```

## Quickstart

```python
from layerscale import LayerScale

client = LayerScale()  # defaults to http://127.0.0.1:8080

resp = client.chat.create(messages=[{"role": "user", "content": "Hello!"}])
print(resp.choices[0].message.content)
```

```python
from layerscale import AsyncLayerScale

async with AsyncLayerScale() as client:
    resp = await client.chat.create(messages=[{"role": "user", "content": "Hello!"}])
```

The base URL falls back to the `LAYERSCALE_BASE_URL` environment variable,
then to `http://127.0.0.1:8080` (the server's default bind).

### Authentication

The server's Ed25519 license key (format `LSK-...`) doubles as the API bearer
token, the binary refuses to start without one. Every route except the CORS
preflight and `/health`/`/healthz` (this includes `/metrics`) requires
`Authorization: Bearer <LSK-...>` (the server also accepts `x-api-key`) and
answers 401 `authentication_error` without it; the WebSocket upgrade is
refused with a plain-HTTP 401 before the handshake. Pass the key as the
`api_key` constructor option, or let it fall back to the `LAYERSCALE_API_KEY`
env var, then `LAYERSCALE_LICENSE_KEY` (the server's own canonical variable
for the key). The client sends it as `Authorization: Bearer <key>` on every
request, including the WebSocket handshake; see
[.env.example](.env.example).

## Endpoints

| Client call | Endpoint |
|---|---|
| `client.chat.create(...)` / `client.chat.stream(...)` | `POST /v1/chat/completions` |
| `client.messages.create(...)` / `client.messages.stream(...)` | `POST /v1/messages` |
| `client.models.list()` / `client.models.retrieve(id)` | `GET /v1/models`, `GET /v1/models/{id}` |
| `client.health.check()` | `GET /health` |
| `client.metrics.get()` | `GET /metrics` (Prometheus text) |
| `client.sessions.create/list/get/delete` | `POST/GET /v1/sessions`, `GET/DELETE /v1/sessions/{id}` |
| `client.sessions.push(id, data)` | `POST /v1/sessions/{id}/push` |
| `client.sessions.generate(id, ...)` / `generate_stream(id, ...)` | `POST /v1/sessions/{id}/generate` |
| `client.sessions.flash/list_flash/unflash` | `POST/GET /v1/sessions/{id}/flash`, `DELETE .../flash/{fid}` |
| `client.sessions.events(id)` | `GET /v1/sessions/{id}/events` (SSE) |
| `client.sessions.ws(id)` | `GET /v1/sessions/{id}/ws` (WebSocket) |

Sampling defaults everywhere: `temperature` **0.6**, `top_p` **0.9**,
`max_tokens` **512**. Unsupported parameters are refused by the server with a
400 naming the parameter, the client never forwards fields you didn't pass.

## Chat (OpenAI-compatible)

```python
resp = client.chat.create(
    messages=[
        {"role": "system", "content": "Be brief."},
        {"role": "user", "content": "What's the AAPL price?"},
    ],
    tools=[{"type": "function", "function": {
        "name": "get_stock_price",
        "parameters": {"type": "object", "properties": {"symbol": {"type": "string"}}},
    }}],
    tool_choice="auto",   # only "auto" and "none" are supported
)
choice = resp.choices[0]
if choice.finish_reason == "tool_calls":
    for call in choice.message.tool_calls or []:
        print(call.function.name, call.function.arguments)  # arguments is a JSON string
```

Notes:

- `model` is optional and **ignored**, the response always carries the
  server's model name.
- On prose replies the `tool_calls` key is absent; with tool calls, `content`
  is `None` unless there is visible text.
- `usage.prompt_tokens_details.cached_tokens` appears only when > 0 (session
  turns), it is modeled as optional, never zero-defaulted.
- LayerScale extensions: `session_id` (bind the turn to a session),
  `ignore_eos`, `seed`, `chat_template_kwargs`.

### Chat streaming, SSE dialect A

Untyped `data:` frames terminated by `data: [DONE]` (consumed by the client).
Chat streams send **no heartbeats**, and no bytes at all during prefill;
the only stream with `: heartbeat` comments is the session events stream
(the parser drops SSE comments defensively everywhere):

```python
for chunk in client.chat.stream(
    messages=[{"role": "user", "content": "Hello!"}],
    stream_options={"include_usage": True},
):
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    elif chunk.usage is not None:
        print(f"\n{chunk.usage.total_tokens} tokens")  # final usage chunk, choices == []
```

Tool calls stream as exactly two chunks per call: an opener with
`id`/`type`/`name` and empty `arguments`, then the full arguments string
(never split). With `stream_options={"include_usage": True}` every chunk
carries `usage: null` until the final usage chunk (which has `choices: []`).

## Messages (Anthropic-compatible)

```python
resp = client.messages.create(
    messages=[{"role": "user", "content": "Weather in Paris?"}],
    max_tokens=256,                      # required by the server
    system="Be brief.",                  # string or a list of text blocks
    tools=[{"name": "get_weather", "input_schema": {"type": "object"}}],
    tool_choice={"type": "auto"},        # only "auto" and "none"
)
for block in resp.content:
    if block.type == "text":
        print(block.text)
    elif block.type == "tool_use":
        # input is the parsed JSON arguments, or the raw string if the
        # model emitted invalid JSON
        print(block.name, block.input)
```

Notes: `top_k` and `thinking` are refused by the server; `tool_choice`
`{"type": "any"}` / `{"type": "tool", ...}` are refused;
`usage.cache_read_input_tokens` appears only when > 0. A `tool_use` block's
`input` is usually a parsed JSON object, but falls back to the raw arguments
string when the model's arguments were not valid JSON, guard with
`isinstance(block.input, dict)` before indexing into it. Inside a
`tool_result`'s content array every part must carry an explicit
`type: "text"` (unlike top-level blocks, where it defaults), a part without
one is a 400. Errors from this endpoint use the Anthropic envelope
(`{"type": "error", "error": {...}}`), the exception classes parse both
envelopes.

### Messages streaming, SSE dialect B

Typed `event:` + `data:` frames, **no `[DONE]` sentinel**:

```python
for event in client.messages.stream(messages=[...], max_tokens=256):
    if event.type == "content_block_delta" and event.delta.type == "text_delta":
        print(event.delta.text, end="")
```

Event order: `message_start` → (`content_block_start` / `content_block_delta`
/ `content_block_stop`)* → `message_delta` → `message_stop`. The
`message_delta` usage includes `input_tokens` (unlike Anthropic's real API).
A mid-stream failure arrives as an `event: error` frame and raises
`APIStreamError`.

## Health

`client.health.check()` never raises:

```python
health = client.health.check()
print(health.ok, health.status)          # True/False, 200 | 503 | 0 (connection error)
if health.body:
    print(health.body.status, health.body.sessions)  # "ok" | "unavailable", counters
```

Health lives at `GET /health` (also `/healthz`), **not** under `/v1`.

`client.metrics.get()` returns the server's Prometheus metrics
(`GET /metrics`, `text/plain; version=0.0.4`) verbatim as a string. Unlike
`/health`, the route is not exempt from license bearer auth, a scraper
needs the key too.

## Sessions

Sessions are explicit, durable, and shared by both chat surfaces via
`session_id`.

```python
session = client.sessions.create(id="ticker-feed", window=4096)  # all fields optional

# wait=True polls until the pushed batch is ingested and KV-resident;
# the push-free alternative is watching the events stream / WS `data_updated`.
client.sessions.push(session.id, ["AAPL 187.44 +0.5%", "MSFT 420.21 -0.1%"], wait=True)

answer = client.sessions.generate(session.id, prompt="What's trending?", max_tokens=32)
print(answer.text, answer.usage.prompt_tokens)

client.sessions.delete(session.id)
```

- `push` takes raw text, a string or a non-empty list of strings. The ack is
  immediate; backpressure never blocks or errors (overflow drops the oldest
  unprocessed entries, counted in `dropped`). `wait=True` (with
  `wait_timeout=30.0`, `wait_interval=0.1`) is a client-side convenience
  that polls `get()` until the server's residency predicate holds —
  `pending == 0`, `data_version` stamped past the ack's, and
  `kv_tokens + 1 >= data_end` (`pending == 0` alone is not proof of
  ingestion), and raises `TimeoutError` on the deadline; the server never
  blocks on push.
- `generate` is **prompt-only** (no messages/tools) and ephemeral, the Q/A
  tail is evicted by the next durable change. Optional `fast_answer`
  (list of candidate strings) and `gap_threshold` enable the speculative
  ready-position exit; hits come back with `speculative: True` /
  `logit_gap`, flash-cache hits with `flash: True` / `flash_id` /
  `confidence`, both with zero usage and no `total_tokens`.
- `delete` raises `ConflictError` (409) while a foreground request is in
  flight, and may take up to ~2 s while background ingest is cancelled.

### Streaming generate

Dialect A: `{"text": ...}` frames, then a `{"done": true, ...}` result frame,
then `data: [DONE]` (consumed by the client):

```python
from layerscale.types import GenerationDone

for chunk in client.sessions.generate_stream(session.id, prompt="Trend?"):
    if isinstance(chunk, GenerationDone):
        print(f"\n[{chunk.finish_reason}] v{chunk.data_version}")
    else:
        print(chunk.text, end="")
```

### Flash queries

Standing questions re-evaluated in the background after every ingested batch
(at most 20 per session; duplicates are a 409):

```python
fq = client.sessions.flash(session.id, "Is the market bullish or bearish?", max_tokens=16)
listed = client.sessions.list_flash(session.id)   # .data, .data_version
client.sessions.unflash(session.id, fq.id)
```

A flash query's `value` / `data_version` / `confidence` / `evaluated_at`
fields exist only once an answer has been computed; `fresh` means the answer
matches the current `data_version`.

### Events stream (SSE dialect B)

```python
for event in client.sessions.events(session.id):
    if event.type == "connected":       # always first: session_id, data_version, flash_queries
        ...
    elif event.type == "flash_ready":   # replayed cached answers, then live changes
        print(event.query, "→", event.value)
    elif event.type == "data_updated":  # every durable settle
        print("v", event.data_version, "tokens", event.tokens, "pending", event.pending)
```

Undecodable frames are dropped (lenient decoding); heartbeat comments
(`: heartbeat`, sent after 15 s of event silence, this is the only stream
that has them) are ignored. When the stream ends (session deleted, subscriber
too far behind, shutdown), reconnect and resync from `connected` + replay.

### WebSocket

```python
from layerscale import WsFlashReady, WsPushAck

with client.sessions.ws(session.id) as ws:   # async: `async with`, `await ws.push(...)`
    ws.push(["line one", "line two"])        # → push_ack event
    ws.ping()                                # JSON ping → pong event
    for event in ws:                         # first event is always `connected`
        if isinstance(event, WsPushAck):
            print("acked at v", event.data.data_version)
        elif isinstance(event, WsFlashReady):
            print(event.data.query, "→", event.data.value)
```

The server sends a protocol ping (`hb`) every 15 s; the `websockets` library
answers pings automatically, so the socket stays alive without extra code.
JSON `error` events are non-fatal (the connection stays open). `close()`
sends the JSON `{"type": "close"}` message, then the WebSocket close frame.

The upgrade request carries the client's configured headers (Authorization,
`default_headers`, User-Agent) just like every HTTP request. A refused
upgrade, the server answers plain HTTP instead of `101`, raises the same
typed errors as the HTTP surface: `AuthenticationError` for a missing or
invalid license key (401, refused before the handshake), `NotFoundError` for
an unknown session (404), `BadRequestError` for a malformed path (400).

The server refuses any inbound frame over 16 MiB before reading its payload
and tears the connection down, no JSON `error` event, the socket just
drops, so split larger pushes into smaller batches.

## Errors

```
LayerScaleError
├── APIError
│   ├── APIConnectionError
│   │   └── APITimeoutError
│   ├── APIStreamError          # mid-SSE error frame
│   └── APIStatusError          # .status_code, .message, .type, .body, .response
│       ├── BadRequestError           (400)
│       ├── AuthenticationError       (401)
│       ├── PermissionDeniedError     (403)
│       ├── NotFoundError             (404)
│       ├── ConflictError             (409)
│       ├── UnprocessableEntityError  (422)
│       ├── RateLimitError            (429)
│       └── InternalServerError       (5xx)
```

Both server envelopes are parsed: `{"error": {"message", "type"}}` on every
route, and `{"type": "error", "error": {"type", "message"}}` on
`POST /v1/messages`.

### Retries

- **Only GET requests retry**, on connection errors, 408, and the
  body-budget 503 (`"server is holding too many request bodies"`), with
  exponential backoff (`max_retries`, default 2). Other 503s (e.g. the
  not-ready `/health` answer or a wedged session) are not transient and are
  **not** retried.
- **POST and DELETE never retry.** A retried DELETE whose first attempt
  succeeded but whose response was lost would surface as a spurious 404.
- **409 is never retried**, on this server it means duplicate-create or
  session-busy, not a transient conflict.
- Streams never retry.

## Timeouts & per-request options

Default `httpx` timeout: **600 s** total with **5 s** connect. Override per
client (`LayerScale(timeout=30.0)`); `timeout=0` or `timeout=None` disables
the timeout entirely (matching the TypeScript client's `timeoutMs: 0`).
Streaming requests skip the default read timeout, they keep the 5 s connect
bound but the read side is open-ended by design (session event streams idle
between events).

Every resource method also accepts a keyword-only `request_options` mapping
with per-request overrides:

```python
client.models.list(request_options={"timeout": 5.0, "max_retries": 0, "headers": {"X-Trace": "1"}})
```

To cancel an in-flight streaming request, close the `Stream`/`AsyncStream`
(or leave its `with` block), the server notices the disconnect and cancels
generation; for non-streaming calls the `timeout` override is the
cancellation mechanism.

Note the server closes idle kept-alive connections after ~30 s, the client
treats the resulting connection errors as retryable for GET requests.

## Examples

- [`examples/example_http.py`](examples/example_http.py), health, models,
  chat with tool calling, session create/push/generate, flash queries, SSE.
- [`examples/example_ws.py`](examples/example_ws.py), async client, WebSocket
  push and live flash results.
