Metadata-Version: 2.5
Name: smart-turn-livekit
Version: 0.2.1
Summary: LiveKit Agents plugin for Smart Turn end-of-turn detection
Project-URL: Homepage, https://github.com/santhosh-005/smart-turn-livekit
Project-URL: Documentation, https://github.com/santhosh-005/smart-turn-livekit#readme
Project-URL: Issues, https://github.com/santhosh-005/smart-turn-livekit/issues
Project-URL: Source, https://github.com/santhosh-005/smart-turn-livekit
Project-URL: Smart Turn (upstream), https://github.com/pipecat-ai/smart-turn
Project-URL: LiveKit Agents, https://github.com/livekit/agents
Author: Santhosh
License-Expression: BSD-2-Clause
License-File: LICENSE
Keywords: conversational-ai,end-of-turn,endpointing,eot,livekit,livekit-agents,livekit-plugin,onnx,pipecat,realtime,semantic-vad,smart-turn,speech,turn-detection,turn-detector,turn-taking,vad,voice-agent,voice-ai
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: numpy>=1.24
Requires-Dist: onnxruntime>=1.16
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: soundfile>=0.12; extra == 'dev'
Provides-Extra: livekit
Requires-Dist: livekit-agents>=1.0; extra == 'livekit'
Requires-Dist: soxr>=0.3; extra == 'livekit'
Provides-Extra: resample
Requires-Dist: soxr>=0.3; extra == 'resample'
Description-Content-Type: text/markdown

# smart-turn-livekit

**LiveKit Agents plugin for Smart Turn end-of-turn detection.**

[![PyPI](https://img.shields.io/pypi/v/smart-turn-livekit)](https://pypi.org/project/smart-turn-livekit/)
[![Python](https://img.shields.io/pypi/pyversions/smart-turn-livekit)](https://pypi.org/project/smart-turn-livekit/)
[![License](https://img.shields.io/badge/license-BSD--2--Clause-blue)](LICENSE)

A LiveKit Agents plugin that runs [Smart Turn](https://github.com/pipecat-ai/smart-turn) models to detect when a user has finished speaking based on audio prosody. It handles the mel front-end, the ONNX session, resampling, per-model language gating, and the threading, so any checkpoint plugs into an `AgentSession` in one line.

Works with upstream's `smart-turn-v3`, a fine-tune, or your own ONNX file. CPU-only, no PyTorch.

```bash
pip install 'smart-turn-livekit[livekit]'
```

## Quickstart

```python
from livekit.agents import AgentSession
from livekit.plugins import silero
from smart_turn_livekit import SmartTurnDetector

session = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.25),   # must be >= 0.25
    turn_handling={
        "turn_detection": SmartTurnDetector(),         # smart-turn-v3
        "endpointing": {"min_delay": 0.3, "max_delay": 2.5},
    },
)
```

Weights download from Hugging Face on first use and are cached. Nothing to
configure, nothing bundled in the wheel.

Inspect each decision with `on_prediction`:

```python
SmartTurnDetector(
    on_prediction=lambda p: print(
        f"end of turn {p.probability:.1%} -> "
        f"{'complete' if p.complete else 'incomplete'} ({p.inference_s * 1000:.0f} ms)"
    )
)
```

```
end of turn 1.2%  -> incomplete (55 ms)
end of turn 94.5% -> complete   (58 ms)
```

## Models

```python
SmartTurnDetector()                                  # smart-turn-v3, default
SmartTurnDetector(model="smart-turn-tamil-tiny")     # a fine-tune
SmartTurnDetector(model_path="/opt/my-model.onnx")   # your own
```

| key | languages | size | accuracy |
|---|---|---|---|
| **[`smart-turn-v3`](https://huggingface.co/pipecat-ai/smart-turn-v3)** *(default)* | [multi](https://github.com/pipecat-ai/smart-turn#features) | 8.7 MB | 92.63% on upstream's benchmark |
| [`smart-turn-v3-fp32`](https://huggingface.co/pipecat-ai/smart-turn-v3) | [multi](https://github.com/pipecat-ai/smart-turn#features) | 32 MB | unquantised reference graph |

#### Language-specific fine-tunes

| key | language | size | accuracy |
|---|---|---|---|
| [`smart-turn-tamil-tiny`](https://huggingface.co/santhosh-005/smart-turn-tamil) | ta | 8.7 MB | 83.71% on real Tamil telephone speech |
| [`smart-turn-tamil-base`](https://huggingface.co/santhosh-005/smart-turn-tamil) | ta | 21 MB | 86.13% on the same |

`MODEL_REGISTRY` is the only place that knows one model from another, so adding
a checkpoint is a dict entry. Anything not listed still runs via `model_path=`.

## Latency

| threads | p50 | p95 |
|---|---|---|
| 1 | 84.5 ms | 91.9 ms |
| 2 | 53.1 ms | 61.7 ms |

`smart-turn-v3`, batch 1, inference only, idle Intel i5-12450H. **Measure on
your own box** — `python -m smart_turn_livekit.bench`. This number lands
directly on the gap between the user finishing and the agent replying, so
budget it against `min_delay`. The bench refuses to print a table without
warning you when machine load is high, because a benchmark taken under load
once produced a wrong shipping recommendation here.

## Gotchas

- **`min_silence_duration >= 0.25`.** LiveKit will not request a prediction
  below `MIN_SILENCE_DURATION_MS + 50` ms and raises at session start if the
  VAD is faster. A floor, not a ceiling — it fails loudly.
- **The language must be one the model claims, or unknown.**
  `supports_language()` returns `False` otherwise, using the languages the
  model was *benchmarked* on. This one fails **silently** — the detector is
  skipped and the agent falls back to fixed timing. Override with
  `languages=(...)`.

## Threshold

Under LiveKit the threshold picks *how long to wait*, not whether to end the
turn: below it the session takes `max_delay` instead of `min_delay`. So a false
`complete` costs a shorter pause rather than an interruption — a softer knob
than the same number under Pipecat.

```python
SmartTurnDetector(threshold=0.72)
SmartTurnDetector(operating_point="polite")   # if the model ships one
```

0.5 is the default threshold. Some language-specific fine-tunes offer predefined operating points (like `"polite"` or `"balanced"`) that are calibrated on held-out evaluation sets to provide a different tradeoff between interruption rate and latency.

## Outside LiveKit

The detector is a thin adapter over a plain class you can call directly:

```python
from smart_turn_livekit import SmartTurn

st = SmartTurn()                  # smart-turn-v3
st.probability(wave_16k)          # P(speaker has finished), 0.0 .. 1.0
st.is_complete(wave_16k)          # bool, against st.threshold
```

16 kHz mono float, last 8 seconds only. Other sample rates need `soxr`.

**Pipecat** needs no adapter — its built-in analyzer takes any Smart Turn ONNX,
and this package can still resolve the file for it:

```python
from smart_turn_livekit import resolve_model
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path=str(resolve_model()))
```

**Pinned or air-gapped:**

```python
SmartTurnDetector(model_path="/opt/models/smart-turn.onnx")
SmartTurnDetector(local_only=True)                  # cached weights only
```

or `SMART_TURN_MODEL=/path/to.onnx`, or `HF_HUB_OFFLINE=1`. An explicit
`model_path` claims **no** language until you pass `languages=(...)`.

## How it works

LiveKit exposes two turn-detector protocols — one reads text, one reads audio
— as plain `typing.Protocol` classes with no base class to import. This
package implements the audio one structurally, so it's accepted by duck typing
alone; there is nothing to register.

One ONNX code path runs every registered model, because every Smart Turn
checkpoint shares the same signature: `input_features (batch, 80, 800)` in, a
`logits` output that is already a sigmoid out. Applying a second sigmoid to
that output gives confident, plausible, wrong answers, so `SmartTurn.probability`
checks the output is in `[0, 1]` and raises rather than return a
silently-broken score.

Inference runs in a single-thread executor so a ~10–30 ms ONNX call never
stalls the event loop. Construction resolves LiveKit's event class and runs one
dummy inference up front (`warmup=True`, default) — skipping that would put a
~1.7 s import penalty on the first prediction of the first call, a cost
LiveKit's own CLI normally hides by paying it earlier.

The mel front-end matches `WhisperFeatureExtractor(chunk_length=8)` and is
checked against Pipecat's own extractor to <1e-3.

## Tests

```bash
pip install -e '.[dev,livekit]'
pytest                                  # 22 tests; 3 skip without the research corpus
python -m smart_turn_livekit.bench
```

Checks features against Pipecat's extractor, the LiveKit Protocols by
`isinstance`, and that **every registered model loads and returns a
probability** — including upstream's, which we do not control.

## Limitations

- **Not a VAD.** It answers "did they finish?", not "is anyone talking?" You
  still need Silero upstream.
- **Audio-only, no language token.** Nothing stops any of these models running
  on any language — `supports_language()` reflects what was *measured*, not
  what is possible.
- **Written against `livekit-agents` 1.7.0.** `TurnDetectionEvent` and both
  turn detector Protocols are private (`_`-prefixed) there, so treat a
  minor-version bump as something to re-check. The test suite fails loudly if
  the shape moves.

## Author

Built and maintained by **Santhosh** ([@santhosh-005](https://github.com/santhosh-005)).

## Licence

**This package** is BSD-2-Clause. It bundles Whisper's mel filterbank (MIT,
OpenAI).

**The weights** are downloaded, not distributed here, and licensed separately.
`smart-turn-v3` is BSD-2-Clause by Daily.
