Metadata-Version: 2.4
Name: llm-fingerprinter
Version: 0.4.2
Summary: Black-box LLM fingerprinting system for model identification
Author-email: litemars <maxmassi12@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/litemars/LLM-Fingerprinter
Project-URL: Repository, https://github.com/litemars/LLM-Fingerprinter
Project-URL: Issues, https://github.com/litemars/LLM-Fingerprinter/issues
Keywords: llm,fingerprinting,model-identification,machine-learning,nlp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: nltk>=3.8.0
Requires-Dist: requests>=2.28.0
Requires-Dist: tenacity>=8.0.0
Requires-Dist: click>=8.0.0
Requires-Dist: joblib>=1.1.0
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Provides-Extra: gemini
Requires-Dist: google-genai>=0.1.0; extra == "gemini"
Provides-Extra: all
Requires-Dist: openai>=1.0.0; extra == "all"
Requires-Dist: google-genai>=0.1.0; extra == "all"
Requires-Dist: httpx>=0.24.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# LLM Fingerprinting System

[![PyPI version](https://badge.fury.io/py/llm-fingerprinter.svg)](https://pypi.org/project/llm-fingerprinter/)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A black-box fingerprinting system that estimates an LLM's model family (GPT, LLaMA, Mistral, etc.) by analysing response patterns across 31 prompts. It can return `unknown` when the observations do not support a known family. General identification of fine-tunes, distillations and their base models has not yet been validated.

**Note: `config.py` lists configured families; recognition requires representative training data. Backend support alone does not provide it.**

A pre-trained classifier is bundled inside the package (`llm_fingerprinter/model/`) and used automatically.

<img src="img/gpt.png" width="400" height="400" alt="GPT">

---

## How It Works

Fingerprinting runs in three sequential layers:

1. **31 prompts** across 3 layers (discriminative → behavioral → stylistic):
   - *Discriminative* (11): Identity, knowledge cutoff, architecture, reasoning — most separating power
   - *Behavioral* (7): Safety boundaries, jailbreak resistance, honesty, policy handling
   - *Stylistic* (13): Formatting, creativity, constraint following, default voice

2. **Feature extraction** per response: 384 embedding dimensions + 12 linguistic features + 6 behavioral features. Responses are averaged within each layer, giving **402 dimensions per layer** and **1206 dimensions per complete fingerprint**.

3. **Embedding rebalancing**: Per-layer PCA retains up to 64 embedding components, limited by the numerical rank of the original observations. PCA, scaling and optional global PCA are fitted **before synthetic augmentation**, using only the training observations. A shared scale within each embedding block preserves its geometry without amplifying weak components. Constant engineered features are excluded, so the working dimension depends on the training data.

4. **Ensemble evidence**: Random Forest (45%) + SVM (45%) + MLP (10%) provide a supporting family estimate and disagreement diagnostics.

5. **Shared identification policy**: Family templates decide the accepted family using cosine distance, ambiguity and distance rejection. The CLI and Python API use this same policy. An unknown family stays `unknown` even if the ensemble or an optional model-version template suggests a match. A version estimate is accepted only when it agrees with an accepted family. Displayed template scores are **similarities, not calibrated probabilities**.

6. **Early stopping** (opt-in): `identify --early-stop <threshold>` can skip remaining layers using the ensemble's provisional score. A fingerprint with skipped layers returns `unknown` from the final identification policy; full-fingerprint templates cannot validate a partial observation. The default, `--early-stop 0`, runs the full suite.

7. **Collection quality gates**: Empty responses count as failed queries. Every attempted layer must meet its coverage requirement: 60% for identification and 90% for training collection. Embedding failures, incorrect feature dimensions, nonfinite values and zero embedding blocks prevent fingerprint creation. Training imports also validate the available collection metadata and feature schema, skipping invalid or explicitly incomplete records with a logged reason. Only layers deliberately skipped by early stopping can be padded, and those records are excluded from training imports.

Newly saved fingerprints retain all successfully collected prompts and raw responses, the feature schema, prompt-suite hash, generation settings, layer coverage and query counts. Historical files remain usable when the checks supported by their recorded data pass; missing historical provenance cannot be reconstructed.

---

## Supported Backends

| Backend | Description | API Key Required |
|---------|-------------|------------------|
| `ollama` | Local Ollama instance | ❌ No |
| `ollama-cloud` | Ollama Cloud API | ✅ `OLLAMA_CLOUD_API_KEY` |
| `openai` | OpenAI API (or compatible) | ✅ `OPENAI_API_KEY` |
| `gemini` | Gemini API | ✅ `GEMINI_API_KEY` |
| `deepseek` | DeepSeek API | ✅ `DEEPSEEK_API_KEY` |
| `custom` | **Any HTTP-based LLM API** | Optional |

### About the Custom Backend

The **custom backend** is the most flexible option — use it with:

- Proprietary LLM APIs not natively supported
- Self-hosted LLMs behind HTTP endpoints
- API proxies and gateways
- Any HTTP-based LLM service

All you need is an HTTP request template file. See examples in `./example/`.

Responses can use JSON, SSE, NDJSON or `Content-Type: text/plain`. Streamed text fragments preserve spaces and newlines. Structured error events, malformed framing and incomplete streams raise an error instead of becoming fingerprint text. For a known plain-text endpoint that omits `Content-Type`, opt in through the Python API with `CustomClient(request_file="request.txt", allow_plain_text=True)`; the CLI requires the endpoint to return the appropriate content type.

---

## Installation

### From PyPI

```bash
# Core package
pip install llm-fingerprinter

# With OpenAI support
pip install llm-fingerprinter[openai]

# With Gemini support
pip install llm-fingerprinter[gemini]

# With all backends
pip install llm-fingerprinter[all]
```

## Quick Start

### 1. Identify a Model (Pre-trained Classifier)

```bash
# Local Ollama
llm-fingerprinter identify -b ollama --model llama3.2

# OpenAI
export OPENAI_API_KEY="your-key"
llm-fingerprinter identify -b openai --model gpt-4o-mini

# Custom endpoint
llm-fingerprinter identify -b custom -r ./custom_request.txt
```

### 2. Train Your Own Classifier

```bash
# Step 1: Generate training fingerprints for each family
#         Temperature is automatically varied across simulations for diversity
llm-fingerprinter simulate -b ollama --model llama3.2 --family llama --num-sims 5
llm-fingerprinter simulate -b openai --model gpt-4o-mini --family gpt --num-sims 5

# Step 2: Train and save the ensemble plus matching family templates
llm-fingerprinter train

# Step 3: Optionally enable estimates of already represented model versions
llm-fingerprinter build-model-templates

# Step 4: Identify unknown models
llm-fingerprinter identify -b ollama --model some-unknown-model
```

`train` saves family templates both inside the classifier artifact and as the matching external template store. Python's `LLMFingerprinter.identify()` uses the embedded templates by default; callers can inject `family_templates` and `model_templates` explicitly. The CLI loads the configured external stores when present, allowing `build-templates` and `add-family` updates to take effect. Use the same stores in Python to reproduce those updated decisions.

Older classifier artifacts keep their original preprocessing when loaded. Run `train` to obtain the corrected representation and embedded family templates; loading an old artifact does not retrain it.

---

### `train --early-stop-variants` — Off by Default

Early stopping produces a *partial* fingerprint: skipped layers are padded with
the mean of completed layers. `--early-stop-variants` adds synthetic examples of
those padding patterns when fitting the ensemble. The representation and family
templates are still fitted only on complete, original observations.

Both early stopping and its training variants are off by default because the
production family decision requires a complete fingerprint. Variants support
experiments with provisional ensemble scores; they do not make partial runs
eligible for an accepted family or version. The effect of variants on a future
partial-observation policy needs its own independent evaluation.

```bash
llm-fingerprinter train                          # full-suite default
llm-fingerprinter train --early-stop-variants    # experimental partial scoring
llm-fingerprinter identify -b ollama --model llama3.2 --early-stop 0
```

`identify --early-stop` warns if the loaded classifier was trained without them.

---

### `train --cross-validate` — Evaluation by Model

Cross-validation groups fingerprints **by model**, so every simulation of one
model stays on the same side of the split. Groups are assigned within each family
to keep every represented family in both training and validation. All transforms,
augmentation and templates are fitted inside the training fold.

This matters because `simulate -n 5` produces five runs of a *single* model at
five temperatures. Those runs can be very similar. Ungrouped evaluation can
therefore overstate performance on unseen models within a known family. The
reported accuracy scores the shared final decision, counting rejected known
samples as unsuccessful. The Python result also includes the ensemble's score
before rejection, accepted coverage and accuracy among accepted results.

The fold count is capped by the family with the fewest distinct models, and the
output says which one:

```bash
llm-fingerprinter train --cross-validate
```

```
   Grouping by model — 21 distinct models across 108 fingerprints
   Folds capped at 2 by family 'llama' (2 distinct model(s))
```

To get more folds — and a less noisy estimate — collect fingerprints for more
*distinct models* within a family, not more simulations of the same one.

The corrected implementation was evaluated on the existing **108 fingerprints
from 21 models**, using the same seven two-fold model-grouped splits as the code
review. Mean raw ensemble family accuracy was **94.84%**; the shared final policy
scored **94.58%**, counting rejection as unsuccessful. In a separate challenge
that excluded each family from training in turn, the shared policy rejected
**105/108** excluded-family samples. These are exploratory results on a small,
reused corpus, not a fresh test set or a general accuracy guarantee.

### What to Collect Next

- Reserve new, independent models and collection sessions as a test set before changing prompts, thresholds or classifiers. Evaluate additional runs of known versions, unseen models within known families, and entirely unknown families separately.
- Broaden family coverage: Gemini is configured but currently has no samples in the reviewed corpus. Add more Llama/Qwen generations and independently sourced fine-tunes with known ancestry before adding many more runs of the existing models.
- Collect the same models across backends and system-prompt settings, and keep raw responses and model/deployment identifiers. This helps distinguish family signal from provider or configuration effects.
- Use the new raw-response and schema records for re-extraction and prompt ablation. Keep held-out data out of feature selection and threshold tuning.

---

### `build-templates` — Build Family Template Classifier

Rebuild the external family template store from valid training fingerprints, for example after changing the collection or the rejection threshold. `train` already creates matching family templates, so this is not a required extra step after training.

```bash
llm-fingerprinter build-templates
```

The template classifier uses cosine distance to the nearest family mean and rejects distant or ambiguous queries. Its decision is authoritative in the shared policy. At least two distinct, usable family templates are required. Rebuilding this external store does not replace templates embedded in an older classifier artifact; Python callers should inject the updated store or retrain.

---

### `build-model-templates` — Build Model-Level Templates

Build optional templates for model versions represented in the collection (e.g. `gpt-4o-mini` vs `gpt-4.1`). A model-version match is accepted only within an accepted, agreeing family; it cannot rescue an unknown family decision or establish a version absent from the store.

```bash
llm-fingerprinter build-model-templates
```

Requires fingerprints that contain `model_name` in their metadata (all fingerprints generated with `simulate` on this version do).

---

### `add-family` — Add a New Family Without Retraining

Add a new model family to the template classifier from a few fingerprint samples, without retraining the full ensemble.

```bash
llm-fingerprinter add-family --model deepseek-chat --family deepseek --num-fps 3 -b deepseek
```

Start with at least 3 valid, complete fingerprints. More independent models and sessions are needed to assess whether the family template generalizes. Existing templates for another family must already be available. The CLI uses the updated external store; Python callers should inject that store explicitly.

---

## Environment Variables

| Variable | Backend | Description |
|----------|---------|-------------|
| `OLLAMA_CLOUD_API_KEY` | ollama-cloud | Ollama Cloud API key |
| `OPENAI_API_KEY` | openai | OpenAI API key |
| `GEMINI_API_KEY` | gemini | Gemini API key |
| `DEEPSEEK_API_KEY` | deepseek | DeepSeek API key |
| `LOG_LEVEL` | all | Logging level (`DEBUG`, `INFO`, `WARNING`) |
| `LLM_FINGERPRINTER_DATA` | all | Override data directory (fingerprints, logs) |
| `LLM_FINGERPRINTER_MODEL` | all | Override the model/templates directory |

### Where data and model artifacts live

| How you run it | Data directory | Model directory |
|----------------|----------------|-----------------|
| From a source checkout | the checkout root | `llm_fingerprinter/model/` |
| Installed from PyPI | `~/Library/Application Support/llm-fingerprinter` (macOS), `$XDG_DATA_HOME/llm-fingerprinter` (Linux), `%LOCALAPPDATA%\llm-fingerprinter` (Windows) | `<data dir>/model` |

These locations are derived from where the package is installed, **never from your
current working directory**. Model artifacts are joblib files, and loading one
executes the code it contains — so an artifact is only loaded from the installed
package, the per-user data directory, or a path you name explicitly via
`LLM_FINGERPRINTER_MODEL` / `LLM_FINGERPRINTER_DATA`. Anything else is refused.

> Upgrading from ≤ 0.4.1? Earlier versions located the data directory by walking
> up from the working directory to the nearest `setup.py`/`.git`, so fingerprints
> could end up inside whatever repository you happened to be in. Those files are
> still on disk; point `LLM_FINGERPRINTER_DATA` at that directory to keep using
> them. The CLI prints a reminder when it spots one.

---

## License

MIT License
