Metadata-Version: 2.5
Name: anyinfer
Version: 0.1.3
Summary: An application-owned hybrid inference runtime for Python, spanning hosted APIs and managed local models.
Project-URL: Homepage, https://github.com/anthturner/AnyInfer
Project-URL: Documentation, https://anyinfer.dev/
Project-URL: Changelog, https://github.com/anthturner/AnyInfer/releases
Project-URL: Issues, https://github.com/anthturner/AnyInfer/issues
Author-email: Anthony Turner <github@anthturner.com>
License-Expression: MIT
License-File: LICENSE
Keywords: anthropic,inference,llama.cpp,llm,ollama,openai,routing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx2>=2.0
Requires-Dist: jsonschema>=4.21
Provides-Extra: all
Requires-Dist: azure-identity>=1.17; extra == 'all'
Requires-Dist: cryptography>=42; extra == 'all'
Requires-Dist: github-copilot-sdk>=0.1; extra == 'all'
Requires-Dist: keyring>=24; extra == 'all'
Requires-Dist: markdown>=3.6; extra == 'all'
Requires-Dist: opentelemetry-api>=1.25; extra == 'all'
Requires-Dist: pyside6>=6.6; extra == 'all'
Requires-Dist: starlette>=0.37; extra == 'all'
Requires-Dist: tiktoken>=0.7; extra == 'all'
Requires-Dist: uvicorn>=0.30; extra == 'all'
Provides-Extra: attest
Requires-Dist: cryptography>=42; extra == 'attest'
Provides-Extra: azure
Requires-Dist: azure-identity>=1.17; extra == 'azure'
Provides-Extra: copilot
Requires-Dist: github-copilot-sdk>=0.1; extra == 'copilot'
Provides-Extra: demo
Requires-Dist: markdown>=3.6; extra == 'demo'
Requires-Dist: pyside6>=6.6; extra == 'demo'
Provides-Extra: dev
Requires-Dist: import-linter>=2.0; extra == 'dev'
Requires-Dist: markdown>=3.6; extra == 'dev'
Requires-Dist: mkdocs-material>=9.5; extra == 'dev'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.25; extra == 'dev'
Requires-Dist: pyside6>=6.6; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-xdist>=3.6; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: keyring
Requires-Dist: keyring>=24; extra == 'keyring'
Provides-Extra: mcp
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.25; extra == 'otel'
Provides-Extra: serve
Requires-Dist: starlette>=0.37; extra == 'serve'
Requires-Dist: uvicorn>=0.30; extra == 'serve'
Provides-Extra: tokenizers
Requires-Dist: tiktoken>=0.7; extra == 'tokenizers'
Provides-Extra: vertex
Requires-Dist: cryptography>=42; extra == 'vertex'
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/anthturner/AnyInfer/main/docs/assets/anyinfer-horizontal-dark.svg">
    <img src="https://raw.githubusercontent.com/anthturner/AnyInfer/main/docs/assets/anyinfer-horizontal-light.svg" alt="AnyInfer" width="480" />
  </picture>
</p>

<p align="center">
  <a href="https://github.com/anthturner/AnyInfer/actions/workflows/ci.yml"><img src="https://github.com/anthturner/AnyInfer/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI" /></a>
  <a href="https://anyinfer.dev/"><img src="https://img.shields.io/badge/docs-anyinfer.dev-2C7A6F" alt="Documentation" /></a>
  <a href="https://github.com/anthturner/AnyInfer/releases/latest"><img src="https://img.shields.io/github/v/release/anthturner/AnyInfer?include_prereleases&label=release&color=E8963C" alt="Latest release" /></a>
  <a href="https://github.com/anthturner/AnyInfer/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-2C7A6F" alt="MIT license" /></a>
  <img src="https://img.shields.io/badge/python-3.11%2B-0B3B3C" alt="Python 3.11+" />
</p>

**An application-owned hybrid inference runtime for Python.** Send one typed request to
hosted providers, routing hubs, existing local servers, or a supervised `llama.cpp`
process. The same core owns routing, structured output, context budgeting and reduction,
telemetry, and shared configuration.

```python
import anyinfer as ai

text = "AnyInfer 0.1 adds embeddings, reranking, and an OpenAI-compatible sidecar."

client = ai.Client(
    [
        ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"),
        ai.ProviderSettings.of("ollama"),
    ]
)
result = client.generate(
    "Summarize this release note:\n" + text, target="anthropic:claude-sonnet-4-5"
)

print(result.text)
print(result.usage.output_tokens, "tokens in", result.timing.total_ms, "ms")
```

Point the same call at a local model by changing one string:

```python
result = client.generate(
    "Summarize this release note:\n" + text, target="ollama:qwen3:8b"
)  # "medium" is a catalog alias
```

Embeddings and reranking are the same client, the same routing, and the same batching —
never a bolted-on provider option:

```python
vectors = client.embed(["first passage", "second passage"], target="ollama:nomic-embed-text")
ranked = client.rerank("which passage answers the question", passages, target="cohere:rerank-v3.5")
```

---

## Why this exists

There are already good libraries for switching between cloud providers, good gateways for
centralizing model traffic, and good local-model servers. If one of those is your whole
problem, you may not need AnyInfer. See
[when a smaller tool is the better boundary](https://anyinfer.dev/why-anyinfer/#when-a-smaller-tool-is-the-better-boundary).

AnyInfer is for applications that need those environments to behave as one runtime. It
keeps provider translation, local process ownership, context preparation, routing, and
result validation inside one testable boundary:

- **One primitive.** A `GenerationRequest` becomes a typed event stream. Non-streaming is
  the drained stream. It is *not* an OpenAI-API clone. The OpenAI dialect is one edge
  format among several.
- **Adapters only translate.** Retry, fallback, health gating, schema validation, repair,
  TTFT measurement, usage normalization, telemetry, and redaction live in the core, once.
- **Structured output is a contract.** A request carrying a schema always returns a
  client-side-validated result, using the strongest mechanism the provider offers
  (grammar → json_schema → json_mode → prompt), with an opt-in bounded repair loop.
- **Embeddings and reranking are inference primitives, not provider options.**
  `client.embed()`/`client.rerank()` are typed, routed, batched, and cost-tracked exactly
  like generation — with a safety rule generation does not need: a fallback that cannot
  be proven to share the primary target's vector space is refused before dispatch, not
  silently served as a wrong-but-plausible vector.
- **Context engineering is part of dispatch.** A provenance-aware budget estimates input,
  output reserve, headroom, and cost before a call. Deterministic reducers fit approved
  corpora to that budget and report exactly what they omitted; hierarchical distillation
  handles material that cannot fit at any fidelity.
- **Capabilities carry provenance.** Every context window, price, and feature flag records
  whether it was catalogued, discovered, probed, or defaulted. Nothing is guessed silently.
- **Local inference is first-class.** Hardware detection, backend selection, llama-server
  supervision and tuning, verified GGUF downloads, and hardware→tier recommendation.
- **Portability is verified behavior, not a provider count.** Contract snapshots and a
  shared conformance suite document what each adapter actually supports and surface dropped
  parameters or degraded mechanisms instead of hiding them.
- **Your integration is testable.** A scripted provider and pytest fixtures ship with the
  library, so your fallback chain, repair budget, and reduction settings have offline tests
  that run in CI with no credentials, including the failures you cannot provoke on demand:
  rate limits, truncated streams, malformed structured answers, refusals, timeouts.
- **A confidentiality story nobody else in this market ships.** Encrypted-at-rest prompt
  templates, a zero-retention orchestration relay, and a portable capability check that
  tells your application whether a box can back local inference with a real
  hardware-attested guarantee — see [confidentiality
  tiers](https://anyinfer.dev/guides/confidentiality-tiers/).
- **Slim by construction.** Mandatory dependencies are `httpx2` and `jsonschema`. Everything
  else is an extra.

## When it is the right layer

Use AnyInfer when your application needs to own a hybrid route, such as a hosted model
with a managed local fallback, or when context fit, structured results, attempt
history, and telemetry must retain the same meaning after the target changes.

Use a simpler provider client when you only need cloud API switching. Use a gateway when
you need organization-wide keys, quotas, spend controls, or an admin plane. Use Ollama,
LM Studio, or LocalAI directly when a dedicated local-model service is the product boundary.

**[Why AnyInfer →](https://anyinfer.dev/why-anyinfer/)** goes through the five capabilities
that are genuinely unusual here, with a dated comparison by category and the commands that
check each claim. The same page argues the other side, naming the tools that are a better
fit when they are.

## Install

```bash
pip install anyinfer                 # core + local lifecycle; runtimes fetched separately
pip install "anyinfer[copilot]"      # GitHub Copilot
pip install "anyinfer[azure]"        # Azure AI Foundry (Entra auth)
pip install "anyinfer[vertex]"       # Vertex service-account authentication
pip install "anyinfer[keyring]"      # credential:// references via the OS vault
pip install "anyinfer[otel]"         # OpenTelemetry bridge
pip install "anyinfer[serve]"        # the OpenAI-compatible HTTP frontend
pip install "anyinfer[demo]"         # the PySide6 pack-in demo app
pip install "anyinfer[all]"
```

Python 3.11+. Windows, macOS, and Linux are all first-class.

### Two commands to a working call

```bash
anyinfer init      # detect what is usable here, write anyinfer.json and starter.py
python starter.py  # run it
```

`init` reports only what it observed: a loopback engine that answered or a credential
variable that is actually set. It writes detected keys as `env://` references rather
than values, so the file it generates is safe to commit. It installs nothing and never
replaces a configuration you already have.

### Try it without credentials

```bash
pip install "anyinfer[demo]"
anyinfer-demo
```

The [pack-in demo app](https://anyinfer.dev/guides/demo-app/) runs offline against
in-process fakes and shows streaming, routing with retry and fallback, structured output,
and the telemetry event stream. Standalone builds for Windows, macOS, and Linux require no Python and are
attached to [every release](https://github.com/anthturner/AnyInfer/releases/latest); see the
[downloads page](https://anyinfer.dev/downloads/).

### Working on AnyInfer

```bash
python workspace.py setup    # install the project and dev extras
python workspace.py check    # every gate CI runs
python workspace.py demo     # launch the demo app
```

`python workspace.py` with no arguments lists every verb. See
[Contributing](https://anyinfer.dev/contributing/).

## Compatibility surface

Provider breadth is compatibility inventory, not the product thesis. AnyInfer ships 20
dedicated adapters plus 86 declarative OpenAI-compatible presets.
**[See the complete inventory →](https://anyinfer.dev/providers/all/)**

The dedicated adapters, each handling provider-specific protocol or discovery behavior:

| Provider | Target prefix | Notes |
|---|---|---|
| OpenAI-compatible | `openai-compat:` | Any server speaking `/chat/completions` |
| OpenAI | `openai:` | Responses API |
| Anthropic | `anthropic:` (alias `claude:`) | Messages API |
| Google Gemini | `gemini:` (alias `google:`) | Native generateContent, thinking levels |
| DeepSeek | `deepseek:` | Reasoning channel, split cache accounting |
| xAI | `xai:` (alias `grok:`) | Provider-reported cost, discovered pricing |
| Google Vertex AI | `vertex:` | Gemini with GCP OAuth/ADC auth |
| AWS Bedrock | `bedrock:` | Converse API, SigV4 or Bedrock API key |
| Cohere | `cohere:` | Native v2 chat API |
| Voyage AI | `voyage:` | Specialist embeddings/rerank; query/document intents |
| Jina AI | `jina:` | Specialist embeddings/rerank; full task vocabulary |
| Text Embeddings Inference | `tei:` | Local embeddings/rerank; retrieval-only, one model per server |
| LM Studio | `lm-studio:` | Native model discovery and residency |
| Azure AI Foundry | `azure-foundry:` | Entra or API key |
| GitHub Copilot | `copilot:` | Includes the `auto` sentinel |
| Microsoft 365 Copilot | `m365-copilot:` | Interactive auth only |
| OpenRouter | `openrouter:` | Rich discovered pricing/context metadata |
| Nebius Token Factory | `nebius:` | Live pricing, context, and quantization discovery |
| Ollama | `ollama:` | Native API, grammar-enforced schemas |
| llama.cpp | `llama-cpp:` | Supervised `llama-server`, loopback-only |
| 86 more, preconfigured | `groq:` `together:` `mistral:` `vllm:` … | [OpenAI-compatible presets](https://anyinfer.dev/providers/presets/) |

See the [complete provider list](https://anyinfer.dev/providers/all/),
the [provider guides](https://anyinfer.dev/providers/) and the
[conformance matrix](https://anyinfer.dev/reference/conformance-matrix/) for exactly what each supports.

Embeddings and/or reranking are live today on OpenAI, Google Gemini, Azure AI Foundry,
Google Vertex AI, AWS Bedrock (Titan), Cohere, Voyage AI, Jina AI, TEI, Ollama, LM Studio,
llama.cpp (against an embeddings-mode server), and four OpenAI-compatible presets
(Together AI, Fireworks AI, DeepInfra, Mistral) — see
[Embeddings and reranking](https://anyinfer.dev/concepts/embeddings/) and the
[semantic-search example](https://anyinfer.dev/examples/semantic-search/).

## Integration paths

| Path | Best for | Entry point |
|---|---|---|
| Python SDK | Python applications that want typed results and the full event stream | `Client` / `AsyncClient` |
| Command-line tool | Shell scripts and one-off prompts | `anyinfer run` |
| OpenAI-compatible sidecar | Non-Python applications and existing OpenAI clients | `anyinfer serve` (pip) or the standalone `anyinfer-serve` bundle |

All three use the same engine and the same versioned JSON configuration format:

```json
{
  "format_version": 1,
  "providers": [{"id": "ollama"}],
  "default_route": ["ollama:qwen3:8b"]
}
```

Load it with `ai.load_config("anyinfer.json")`, pass it to `anyinfer run --config`, or use
it unchanged with the sidecar. See [choosing an integration path](https://anyinfer.dev/guides/#choose-a-path)
and [shared configuration](https://anyinfer.dev/reference/configuration/).

## Documentation

**[anyinfer.dev](https://anyinfer.dev/)** is the published
site, including the generated [SDK reference](https://anyinfer.dev/reference/api/)
and [runnable examples](https://anyinfer.dev/examples/). The same pages are
browsable in-repo from the **[documentation index](https://github.com/anthturner/AnyInfer/blob/main/docs/README.md)**.

Quick links by role:

- **Integrating the Python SDK?** → [Python SDK guide](https://anyinfer.dev/guides/python-sdk/) ·
  [Quickstart](https://anyinfer.dev/guides/quickstart/) · [SDK reference](https://anyinfer.dev/reference/api/)
- **Running the HTTP service?** → [OpenAI-compatible sidecar](https://anyinfer.dev/serve/)
- **Working from a shell?** → [Run a prompt from the shell](https://anyinfer.dev/guides/cli/)
- **Sharing provider and route settings?** → [Configuration](https://anyinfer.dev/reference/configuration/)
- **Letting a coding agent write the integration?** → [Coding agents](https://anyinfer.dev/guides/coding-agents/) ·
  [`llms.txt`](https://anyinfer.dev/llms.txt) · run `anyinfer agents-md >> AGENTS.md`
- **Contributing or writing an adapter?** →
  [Contributor guide](https://github.com/anthturner/AnyInfer/blob/main/CONTRIBUTING.md) ·
  [Provider contracts](https://github.com/anthturner/AnyInfer/blob/main/contracts/README.md)
- **Reporting a vulnerability?** → [Security policy](https://github.com/anthturner/AnyInfer/security/policy)

## Project status

Pre-1.0 and under active development: the public API is settled in shape but may still
move before 1.0. Releases are published to
[PyPI](https://pypi.org/project/anyinfer/) and
[GitHub](https://github.com/anthturner/AnyInfer/releases); the
[release strategy](https://anyinfer.dev/contributing/releasing/) has the
details. The architecture is settled and documented:

- [DESIGN.md](https://github.com/anthturner/AnyInfer/blob/main/DESIGN.md): architecture, module responsibilities, decision rationale, open questions, and risks
- [AGENTS.md](https://github.com/anthturner/AnyInfer/blob/main/AGENTS.md): canonical repository automation instructions

## License

[MIT](https://github.com/anthturner/AnyInfer/blob/main/LICENSE).
