Metadata-Version: 2.4
Name: pipecat-speko
Version: 0.1.0
Summary: Speko speech-to-text and text-to-speech services for Pipecat. One interface across every voice provider, with per-language routing, failover, and a first-byte deadline so a silent upstream fails over instead of hanging the pipeline.
Project-URL: Source, https://github.com/SpekoAI/pipecat-speko
Project-URL: Documentation, https://speko.dev/docs/pipecat
Project-URL: Website, https://speko.dev
Project-URL: Changelog, https://github.com/SpekoAI/pipecat-speko/blob/main/CHANGELOG.md
Author-email: Speko <support@speko.ai>
License: BSD-2-Clause
License-File: LICENSE
Keywords: ai,audio,llm,pipecat,realtime,speko,stt,tts,voice
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.14,>=3.11
Requires-Dist: aiohttp>=3.9
Requires-Dist: loguru>=0.7
Requires-Dist: pipecat-ai<2,>=1.6.0
Requires-Dist: websockets<17,>=12.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# pipecat-speko

Speko speech-to-text and text-to-speech services for
[Pipecat](https://github.com/pipecat-ai/pipecat). Route both speech stages
through one API, with per-language provider ranking and failover, and keep the
rest of your pipeline unchanged. Bring your own LLM.

Tested with Pipecat 1.6.0. Requires `pipecat-ai>=1.6.0,<2`.

Speko is not affiliated with Pipecat. This is a community integration,
maintained by Speko.

## Install

```
pip install pipecat-speko
```

## Use

Two imports and two constructors change. The `Pipeline([...])` list does not.

```python
from pipecat_speko import SpekoPolicy, SpekoSTTService, SpekoTTSService

policy = SpekoPolicy(language="es-MX", optimize_for="latency")

stt = SpekoSTTService(api_key=os.environ["SPEKO_API_KEY"], policy=policy)
tts = SpekoTTSService(
    api_key=os.environ["SPEKO_API_KEY"],
    policy=policy,
    settings=SpekoTTSService.Settings(voice="..."),
)

pipeline = Pipeline([
    transport.input(), stt, user_aggregator, llm, tts,
    transport.output(), assistant_aggregator,
])
```

`examples/foundational.py` is a single runnable file.

## What each service is

| Service | Base class | Method | Frames it produces |
| --- | --- | --- | --- |
| `SpekoSTTService` | `WebsocketSTTService` | `async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame \| None, None]` | Yields `None`; pushes `InterimTranscriptionFrame` then `TranscriptionFrame` from the receive loop |
| `SpekoTTSService` | `TTSService` | `async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame \| None, None]` | `TTSAudioRawFrame` per chunk, bracketed by `TTSStartedFrame` and `TTSStoppedFrame` from the base class |
| `SpekoLLMService` (experimental, see below) | `LLMService` | `async def _process_context(self, context: LLMContext)` | `LLMFullResponseStartFrame`, then `LLMTextFrame` per token, then `LLMFullResponseEndFrame` |

Speech-to-text runs over the gateway's streaming websocket
(`/v1/transcribe/stream`), so transcripts arrive asynchronously and `run_stt`
yields `None` - the same shape the first-party streaming services use.

Synthesis is HTTP (`POST /v1/synthesize`), because that is what the gateway
serves: chunked raw PCM, signed 16-bit, mono, with the chosen provider on the
response headers. `SpekoHttpTTSService` is an alias, for the Pipecat naming
convention. There is no websocket text-to-speech endpoint to wrap.

### SpekoLLMService is experimental and unsupported

`SpekoLLMService` ships in the package but is **not part of the supported
surface in this release**, and constructing it against the default base URL
raises `NotImplementedError` rather than failing on the first inference.

It speaks the OpenAI chat-completions streaming format. `api.speko.dev` does
not serve that surface: the gateway's LLM route is `POST /v1/complete`, and the
live endpoints are `/v1/transcribe`, `/v1/transcribe/stream`, `/v1/synthesize`,
`/v1/voices` and `/v1/routing/preview`. Pointing `base_url` at an
OpenAI-compatible endpoint you control is supported and is what the tests do.

Use Speko for the speech stages and whatever LLM you already run. The example
does exactly that.

## The `policy=` kwarg

`SpekoPolicy` is one object for both halves of a routing decision: the intent
(language, region, objective) and the per-stage provider allowlist. The same
kwarg exists on the LiveKit plugin and means the same thing.

```python
SpekoPolicy(
    language="es-MX",          # BCP-47, validated in the constructor
    region="global",
    optimize_for="latency",    # balanced | accuracy | latency | cost
    tts=["cartesia"],          # per-stage allowlist; one entry pins it
)
```

Each service narrows the allowlist to its own stage, so a `tts=[...]` pin on a
shared policy does not travel on a transcription request. A plain dict works
too - `policy={"language": "es-MX", "optimizeFor": "latency"}` - so a policy can
come straight out of config.

Language tags and objectives are validated where you build the policy, not on
the wire. A typo becomes a `ValueError` at process start instead of a 400 on
turn one of a live call.

## The zero-audio gap, and the first-byte deadline

Pipecat failover is driven by `ErrorFrame` alone. A provider that accepts a
request and then returns nothing produces no `ErrorFrame`, so
`ServiceSwitcherStrategyFailover` never fires. Pipecat issue #5026 is the
production record of exactly that: a websocket text-to-speech service completed
every context with zero audio, and the configured healthy fallback sat unused
through nine of nine calls while the caller heard silence. The merged remedy,
PR #5106, is a timeout rather than detection - `pause_watchdog_timeout_s`
defaults to 3.0 seconds, arms only when `pause_frame_processing=True`, and
force-resumes after the silence has already been heard.

`SpekoTTSService` watches its own clock instead.

- **`first_byte_timeout_s`, default 1.5 seconds.** The budget covers connect,
  headers and the first audio byte. If it elapses, the request is aborted and a
  non-fatal `ErrorFrame` is yielded.
- **Zero-audio completion is also an error.** A clean 200 whose body ends with
  no audio bytes yields an `ErrorFrame` too. This is the case the pause watchdog
  cannot catch, because the context does complete.
- **So is audio that resamples to nothing.** If bytes arrived but no audio frame
  reached the pipeline, that is the same outage from the caller's seat, and it
  gets the same `ErrorFrame`. It is reachable: Pipecat's stream resampler holds
  its warm-up samples back, so a very short utterance delivered in one chunk at
  a rate that does not match the pipeline produces no frames at all.
- **The frame is switchable.** It is non-fatal and carries `processor=self`,
  which is what `ServiceSwitcher.push_frame` checks before it calls
  `handle_error`. The turn then closes normally rather than hanging.
- **No false positives.** A slow-but-alive upstream that delivers its first byte
  inside the budget streams normally; the deadline applies to the first byte
  only, never to the rest of the stream.
- **`first_byte_timeout_s=None`** disables it and restores stock behaviour. The
  test suite keeps a case for this, and it shows what you lose: a silent
  upstream then produces neither audio nor an error.

1.5 seconds is deliberately shorter than Pipecat's 3.0. Anything longer is
audible dead air.

`SpekoLLMService` has the same guard as `first_token_timeout_s`, defaulted to
`None` (off) because a slow first token is more often a long prompt than a dead
provider. Set it if you would rather fail over.

## Interruption

`InterruptionFrame` is a `SystemFrame`, so it bypasses the queues. The base
class clears text aggregation, stops the audio-context task and drops queued
audio. `SpekoTTSService` adds one thing: it closes the in-flight HTTP response
immediately, so the upstream stops generating audio nobody will hear. Without
that, a barge-in still pays for the rest of the utterance and holds the
connection while it drains.

`run_tts` never swallows `CancelledError` - the base class needs the
cancellation to unwind the turn - and the response is closed in a `finally`, so
a cancelled or failed synthesis cannot leak a socket.

## Metrics

Every service returns `True` from `can_generate_metrics()`. Turn metrics on with:

```python
PipelineParams(enable_metrics=True, enable_usage_metrics=True)
```

- **Text-to-speech** reports TTFB stopped on the first audio byte off the
  socket, rather than when the frame reaches the audio context, so the number is
  the upstream's and not the queue's. Usage metrics are reported per synthesis.
- **The LLM** reports TTFB stopped on the first streamed token, processing time
  around the whole completion, and token usage from the stream's usage event.
- **Speech-to-text** reports TTFB the way Pipecat defines it for the modality:
  from the end of user speech to the final transcript, handled by the base class.

Per-request attribution survives routing. `SpekoTTSService.route` exposes the
provider, model, failover count, gateway-measured first-byte milliseconds and
benchmark snapshot id from the `X-Speko-*` response headers, and
`SpekoSTTService.provider` exposes the provider the gateway selected.

## Turn taking

Smart Turn is fed by transport audio, not by speech-to-text, so this service
cannot break it. It can make a turn look slow: the stop strategy reads
`ttfs_p99_latency` off `STTMetadataFrame` as its post-prediction wait budget,
and a service that declares nothing inherits Pipecat's 1.0 second default.

`SpekoSTTService` declares nothing on purpose. Speko routes across providers, so
one figure would be a guess, and this package does not ship guessed numbers.
Measure your own policy and pass it:

```python
SpekoSTTService(api_key=..., ttfs_p99_latency=0.42)
```

## Latency

A service is one `FrameProcessor` in the chain: one queue hop and a task wakeup
per frame, sub-millisecond, and removable with `enable_direct_mode`. What
remains is the network leg to the gateway, and it is additive to
time-to-first-byte. This package forwards the first byte as it arrives, with no
buffering, re-chunking or re-encode, which is the only part it controls.

Speko publishes no overhead figure, because no measured wrapper benchmark
exists. Measure it against a direct-provider service in your own deployment
before you rely on a number.

## Development

```
pip install -e ".[dev]"
pytest
```

The tests run against a local mock gateway (`tests/mock_gateway.py`) that
implements the real wire protocol and can be told to fail the way production
does: a socket that accepts a request and says nothing, a 200 with an empty
body, a slow first byte, a mid-utterance barge-in. Services are driven through
Pipecat's own `run_test` harness, so the frames asserted on are the frames a
real pipeline would see.

## Licence

BSD-2-Clause. See `LICENSE`.
