Metadata-Version: 2.4
Name: llm-proxy-gateway
Version: 0.0.4
Summary: Proxy Server that converts llm request format from one provider to another
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: fastapi>=0.135.3
Requires-Dist: langchain-openai>=0.3.28
Requires-Dist: langchain-core>=0.3.72
Requires-Dist: pydantic>=2.12.5
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: uvicorn>=0.35.0

# LLM Proxy

## Overview

`llm-proxy-gateway` is a small FastAPI service that accepts requests in different LLM API formats and converts them into the OpenAI Responses API format.

Right now, the project supports these endpoint styles:

- OpenAI Chat Completions API
- OpenAI Responses API
- Claude Messages API


The upstream OpenAI call is always made with streaming enabled. The proxy can return either a normal response or a streamed response based on the incoming request.

## Installation

1. Make sure you have `uv` installed.
2. Create a `.env` file in the project root or export variables like [startup.sh](https://github.com/rahgadda/llm-proxy-gateway/blob/main/startup.sh).
3. Add your configuration values.
4. Install dependencies.
5. Start the server.

Example `.env` file:

```env
LLM_PROVIDER=OPENAI
API_SERVER_PORT=11434
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4
OPENAI_BASE_URL=https://api.openai.com/v1
```

Install dependencies:

```bash
uv sync
```

Start the server:

```bash
./startup.sh
```

Or run directly:

```bash
uv run llm-proxy-gateway
```

## Sample Request

Health check:

```http
GET http://localhost:11434/health
```

OpenAI Chat Completions request:
```http
POST http://localhost:11434/v1/chat/completions
Content-Type: application/json

{
  "model": "oca/gpt5",
  "stream": false,
  "messages": [
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ]
}
```

OpenAI Responses request:
```http
POST http://localhost:11434/v1/responses
Content-Type: application/json

{
  "model": "oca/gpt5",
  "input": "Reply with the word OK if you can read this.",
  "stream": false
}
```

Claude Messages request:
```http
POST http://localhost:11434/v1/messages
Content-Type: application/json

{
  "model": "oca/gpt5",
  "max_tokens": 128,
  "messages": [
    {
      "role": "user",
      "content": "Hello Claude, reply with a short greeting."
    }
  ]
}
```

You can also use the ready-made examples in [test.http](./test.http).
