Metadata-Version: 2.4
Name: llm-siren
Version: 0.1.1
Summary: SIREN: a lightweight, plug-and-play guard model for LLM harmfulness detection from internal representations.
Author: Yilun Liu, Ye Yuan, Zhenwei Tang, Linfeng Du, Haolun Wu, Ashton Anderson
Author-email: Difan Jiao <difanjiao@cs.toronto.edu>
License: Apache-2.0
Project-URL: Paper, https://arxiv.org/pdf/2604.18519
Project-URL: Models, https://huggingface.co/UofTCSSLab
Project-URL: Source, https://github.com/CSSLab/SIREN
Keywords: llm,safety,guard-model,harmfulness-detection,content-moderation,siren
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.40
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface_hub>=0.20
Dynamic: license-file

# llm-siren

Python library for **SIREN** ([LLM Safety From Within: Detecting Harmful Content with Internal Representations](https://arxiv.org/pdf/2604.18519), ACL 2026 Main).

SIREN is a lightweight guard model for LLM harmfulness detection. It runs on a small frozen LLM backbone, identifies safety neurons across internal layers, aggregates them into a small MLP classifier, and returns a continuous harmfulness score in `[0, 1]`. The trained classifier head is ~12M parameters; no fine-tuning of the backbone is required.

This package (`llm-siren`) provides the runtime. Trained SIREN heads are released as Hugging Face artifacts under [UofTCSSLab](https://huggingface.co/UofTCSSLab).

## Install

```bash
pip install llm-siren
```

## Usage

```python
import torch
from siren_guard import SirenGuard

guard = SirenGuard.from_pretrained(
    "UofTCSSLab/SIREN-Qwen3-0.6B",
    device="cuda",
    dtype=torch.bfloat16,
)

# Score a user prompt
r = guard.score("How can I make a pipe bomb at home?")
print(r.score, r.is_harmful)

# Score a (prompt, response) pair
r = guard.score(
    prompt="How can I make a pipe bomb at home?",
    response="I can't help with that. Building explosive devices is illegal.",
)

# Batch scoring
results = guard.score_batch([
    "What is the capital of France?",
    "Describe in detail how to commit insurance fraud.",
])

# Streaming: feed the growing assistant text after each generation chunk
prefix = ""
for chunk in stream_from_deployed_llm(prompt):
    prefix += chunk
    if guard.score_streaming(prefix, threshold=0.5).is_harmful:
        break

# Tunable threshold (default 0.5, the binary boundary used during training)
r = guard.score(text, threshold=0.1)  # strict
r = guard.score(text, threshold=0.9)  # permissive
```

## Deployment idiom

```python
def safe_generate(user_prompt, deployed_llm):
    if guard.score(user_prompt).is_harmful:
        return DEFAULT_REFUSAL
    response = deployed_llm.generate(user_prompt)
    if guard.score(prompt=user_prompt, response=response).is_harmful:
        return DEFAULT_REFUSAL
    return response
```

The deployed LLM is independent of SIREN. SIREN never touches the deployed model's internals; it scores the same text through its own frozen backbone.

## Available SIREN artifacts

| Artifact | Backbone | Repo |
|---|---|---|
| `SIREN-Qwen3-0.6B` | Qwen3-0.6B | [UofTCSSLab/SIREN-Qwen3-0.6B](https://huggingface.co/UofTCSSLab/SIREN-Qwen3-0.6B) |

More backbones (Qwen3-4B, Llama-3.2-1B) coming soon.

## API

`SirenGuard.from_pretrained(repo_id_or_path, device=None, dtype=torch.bfloat16, cache_dir=None)`
Loads the SIREN classifier head from an HF repo or local path, plus the frozen backbone at the pinned revision recorded in `siren_config.json`.

`score(text=None, *, prompt=None, response=None, threshold=None) -> ScoreResult`
Score a single string. Pass `text=` for raw moderation, or `prompt=`/`response=` for the response-level form (joined with `"\n"` to match the training distribution).

`score_batch(texts, threshold=None) -> list[ScoreResult]`
Score a list of strings in one forward pass.

`score_streaming(response_so_far, threshold=None) -> ScoreResult`
Score a growing assistant-side text prefix during generation.

Each call returns `ScoreResult(score: float, is_harmful: bool, threshold: float)`.

## License

Apache-2.0.

## Citation

```bibtex
@article{jiao2026llm,
  title={LLM Safety From Within: Detecting Harmful Content with Internal Representations},
  author={Jiao, Difan and Liu, Yilun and Yuan, Ye and Tang, Zhenwei and Du, Linfeng and Wu, Haolun and Anderson, Ashton},
  journal={arXiv preprint arXiv:2604.18519},
  year={2026}
}
```
