Metadata-Version: 2.4
Name: tlex-edge
Version: 0.1.0
Summary: T-LEX: Edge-Decoupled LLM Inference - Run 32B models on ANY device!
Project-URL: Homepage, https://github.com/humoticaos/tlex-edge
Project-URL: Documentation, https://humotica.nl/docs/tlex
Project-URL: Repository, https://github.com/humoticaos/tlex-edge
Author-email: HumoticaOS Team <team@humotica.nl>
License: MIT
License-File: LICENSE
Keywords: ai,edge-computing,inference,iot,llm,low-resource,machine-learning,ollama,quantization
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: requests>=2.28.0
Requires-Dist: sqlite-utils>=3.30
Provides-Extra: dev
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: full
Requires-Dist: chromadb>=0.4.0; extra == 'full'
Requires-Dist: fastapi>=0.100.0; extra == 'full'
Requires-Dist: sentence-transformers>=2.2.0; extra == 'full'
Requires-Dist: transformers>=4.30.0; extra == 'full'
Requires-Dist: uvicorn>=0.22.0; extra == 'full'
Provides-Extra: rag
Requires-Dist: chromadb>=0.4.0; extra == 'rag'
Requires-Dist: sentence-transformers>=2.2.0; extra == 'rag'
Provides-Extra: server
Requires-Dist: fastapi>=0.100.0; extra == 'server'
Requires-Dist: uvicorn>=0.22.0; extra == 'server'
Description-Content-Type: text/markdown

# T-LEX Edge

**Edge-Decoupled LLM Inference - Run 32B+ models on ANY device!**

[![PyPI version](https://badge.fury.io/py/tlex-edge.svg)](https://pypi.org/project/tlex-edge/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## The Problem

Running large language models locally requires expensive GPU hardware:

| Model | VRAM Required | Cost |
|-------|---------------|------|
| 7B    | ~14GB         | $500+ |
| 32B   | ~24GB         | $1000+ |
| 70B   | ~40GB         | $2000+ |

Most edge devices (laptops, IoT, drones, phones) can't run these models.

## The Solution: Split-Brain Architecture

**T-LEX** separates inference (GPU server) from decoding (edge device):

```
┌─────────────────────────┐         ┌─────────────────────────┐
│     GPU Server          │         │     Edge Device         │
│  ┌─────────────────┐    │ tokens  │  ┌─────────────────┐    │
│  │ Ollama/OomLlama │────┼────────►│  │ T-LEX Decoder   │    │
│  │ 32B model       │    │         │  │ vocab.db (6MB)  │    │
│  │ 24GB VRAM       │    │         │  │ NO GPU!         │    │
│  └─────────────────┘    │         │  └─────────────────┘    │
└─────────────────────────┘         └─────────────────────────┘
```

## Performance

| Metric | Local 32B | T-LEX (Remote 32B + Edge Decode) |
|--------|-----------|----------------------------------|
| **GPU Required (Edge)** | 24GB VRAM | **None!** |
| **Edge Storage** | 20GB+ | **6 MB** |
| **Generation Speed** | 2 tok/s | **16 tok/s** |
| **Decode Speed** | N/A | **45,000 tok/s** |

**Result: 8x faster with zero GPU on edge!**

## Installation

```bash
# Basic installation
pip install tlex-edge

# With server support (FastAPI)
pip install tlex-edge[server]

# With RAG support (ChromaDB)
pip install tlex-edge[rag]

# Full installation
pip install tlex-edge[full]
```

## Quick Start

### Python API

```python
from tlex import TLexClient, EdgeDecoder

# Connect to remote GPU server
client = TLexClient("http://gpu-server:11434")

# Generate with 32B model - no local GPU needed!
result = client.generate(
    "Explain quantum computing",
    model="humotica-32b",
    max_tokens=100
)
print(result.text)
print(f"Speed: {result.tokens_per_second:.1f} tok/s")

# Streaming output
for chunk in client.stream("Tell me a story"):
    print(chunk, end="", flush=True)
```

### Command Line

```bash
# Generate text
tlex generate "What is AI?" --model qwen2.5:7b --server http://gpu-server:11434

# Interactive chat
tlex chat --model humotica-32b

# List available models
tlex models

# Benchmark decoder
tlex benchmark --vocab qwen_vocab.db
```

### Docker

```bash
# Build
docker build -t tlex-edge .

# Run
docker run -it tlex-edge generate "Hello!" --model qwen2.5:7b

# With docker-compose
docker-compose up -d
docker-compose exec tlex chat
```

## Building Vocabulary Database

The vocab database is all an edge device needs to decode tokens:

```bash
# From command line
tlex vocab Qwen/Qwen2.5-7B-Instruct --output qwen_vocab.db

# From Python
from tlex import build_vocab_db
build_vocab_db("Qwen/Qwen2.5-7B-Instruct", "qwen_vocab.db")
```

Size comparison:
- **Qwen 7B model**: ~14 GB
- **qwen_vocab.db**: ~6 MB (2300x smaller!)

## Server Setup

T-LEX works with any Ollama-compatible server:

```bash
# On your GPU server (P520, etc.)
ollama serve

# Pull models
ollama pull qwen2.5:7b
ollama pull qwen2.5:32b
```

## Architecture

```
              ┌─────────────────────────────────────────────────┐
              │                  GPU SERVER                      │
              │  ┌─────────────────────────────────────────┐    │
              │  │  Ollama / OomLlama                      │    │
              │  │  - Qwen 7B/32B/72B                      │    │
              │  │  - LLaMA 70B                            │    │
              │  │  - Any GGUF model                       │    │
              │  └─────────────────────────────────────────┘    │
              │                      │                           │
              │                      │ HTTP Stream               │
              │                      │ (token chunks)            │
              └──────────────────────┼───────────────────────────┘
                                     │
                    ┌────────────────┴────────────────┐
                    │           NETWORK               │
                    │    (LAN / Internet / I-Poll)    │
                    └────────────────┬────────────────┘
                                     │
┌────────────────────────────────────┴────────────────────────────────────┐
│                           EDGE DEVICES                                   │
│                                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │   Laptop     │  │  Raspberry   │  │    Phone     │  │    Drone     │ │
│  │              │  │     Pi       │  │              │  │              │ │
│  │ vocab.db 6MB │  │ vocab.db 6MB │  │ vocab.db 6MB │  │ vocab.db 6MB │ │
│  │   NO GPU!    │  │   NO GPU!    │  │   NO GPU!    │  │   NO GPU!    │ │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
```

## Use Cases

- **IoT Devices**: Smart home with AI, no cloud dependency
- **Drones**: On-board AI decisions, low latency
- **Mobile Apps**: Full LLM power without draining battery
- **Air-gapped Networks**: Self-hosted inference + edge decode
- **Cost Reduction**: One GPU server, unlimited edge clients

## Part of HumoticaOS

T-LEX is part of the HumoticaOS ecosystem:

- **TIBET**: Trust & provenance for AI actions
- **I-Poll**: AI-to-AI messaging
- **OomLlama**: Native Rust inference engine

**One love, one fAmIly!** 🦙❤️

## License

MIT License - see [LICENSE](LICENSE) for details.

## Contributing

Contributions welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) first.
