Metadata-Version: 2.4
Name: emergent-translator
Version: 1.1.1
Summary: 60x compression efficiency for AI communication through emergent language translation
Author-email: Emergent Language Team <hello@emergentlanguage.ai>
Maintainer-email: Emergent Language Team <hello@emergentlanguage.ai>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://emergentlanguage.ai
Project-URL: GitHub, https://github.com/maco144/emergent-language
Project-URL: Documentation, https://github.com/maco144/emergent-language/wiki
Project-URL: Repository, https://github.com/maco144/emergent-language.git
Project-URL: Bug Tracker, https://github.com/maco144/emergent-language/issues
Project-URL: Changelog, https://github.com/maco144/emergent-language/releases
Project-URL: API Docs, http://149.28.33.118:8001/docs
Project-URL: Live Demo, http://149.28.33.118:8001
Keywords: ai,compression,emergent-language,api,translation,efficiency,machine-learning,natural-language-processing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Communications
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn[standard]>=0.23.0
Requires-Dist: python-multipart>=0.0.6
Requires-Dist: websockets>=11.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: httpx>=0.24.0
Requires-Dist: aiofiles>=23.0.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: openai>=1.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: isort>=5.12; extra == "dev"
Requires-Dist: flake8>=6.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: pre-commit>=3.0; extra == "dev"
Requires-Dist: pyyaml>=6.0; extra == "dev"
Provides-Extra: monitoring
Requires-Dist: structlog>=23.0.0; extra == "monitoring"
Requires-Dist: prometheus-fastapi-instrumentator>=6.0.0; extra == "monitoring"
Provides-Extra: formats
Requires-Dist: pyyaml>=6.0; extra == "formats"
Requires-Dist: msgpack>=1.0; extra == "formats"
Requires-Dist: protobuf>=4.0; extra == "formats"
Requires-Dist: pyarrow>=12.0; extra == "formats"
Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "formats"
Requires-Dist: tomli_w>=1.0; extra == "formats"
Requires-Dist: pymongo>=4.0; extra == "formats"
Requires-Dist: cbor2>=5.0; extra == "formats"
Requires-Dist: openpyxl>=3.1; extra == "formats"
Provides-Extra: examples
Requires-Dist: langchain>=0.0.300; extra == "examples"
Requires-Dist: crewai>=0.1.0; extra == "examples"
Requires-Dist: jupyter>=1.0.0; extra == "examples"
Requires-Dist: matplotlib>=3.0.0; extra == "examples"
Dynamic: license-file

# Emergent Language Translator

**High-performance binary encoding for AI agent communication**

[![PyPI](https://img.shields.io/pypi/v/emergent-translator)](https://pypi.org/project/emergent-translator/)
[![Python](https://img.shields.io/badge/Python-3.9+-brightgreen)](https://pypi.org/project/emergent-translator/)
[![Tests](https://img.shields.io/badge/Tests-535%20passed-green)]()
[![License](https://img.shields.io/badge/License-GPL--3.0-blue)](LICENSE)

Emergent Language Translator compresses structured AI messages into a compact binary format using learned codebooks, common-key dictionaries, and zlib. Batch encoding amortizes header overhead across messages — the more you batch, the better the ratio.

## Quick Start

```bash
pip install emergent-translator
```

```python
from emergent_translator import BatchEncoder

encoder = BatchEncoder()

# Encode a batch of agent messages
messages = [
    {"role": "user", "content": "analyze market trends", "priority": "high"},
    {"role": "assistant", "content": "Starting analysis", "status": "active"},
    {"role": "system", "content": "Agent coordinator online", "version": "1.0"},
]

result = encoder.encode_batch(messages)
print(f"{len(messages)} messages: 226 bytes JSON -> {len(result.payload)} bytes binary")
# 3 messages: 226 bytes JSON -> 141 bytes (38% reduction)

# Perfect round-trip reconstruction
decoded = encoder.decode_batch(result.payload)
assert decoded == messages
```

## Compression Results

The batch encoder uses a binary wire format with common-key/value dictionaries and zlib compression. Efficiency improves with batch size:

| Workload | JSON Size | Encoded Size | Reduction |
|----------|-----------|-------------|-----------|
| 3 agent messages | 226 bytes | 141 bytes | 38% |
| 10 agent messages | 750 bytes | 112 bytes | **85%** |
| 50 agent messages | 4,880 bytes | 286 bytes | **94%** |

Encoding speed: sub-millisecond (0.2ms typical).

## Binary Format

All payloads start with magic bytes `\xE7\xB0` followed by a version byte:

```
v1/v2: MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + PAYLOAD + CRC32(4)
v3:    MAGIC(2) + VERSION(1) + COUNT(2) + FLAGS(1) + CB_VERSION(2) + CB_LEN(2) + [CODEBOOK] + PAYLOAD + CRC32(4)
```

Common keys (`role`, `content`, `action`, `status`, `priority`, ...) and values (`user`, `assistant`, `system`, `high`, `low`, ...) are encoded as single-byte tokens. Remaining data is zlib-compressed.

## Adaptive Codebooks

The static dictionaries cover common AI communication patterns. For domain-specific traffic, train a codebook that learns your most frequent keys and values:

```python
from emergent_translator import AdaptiveCodebook, BatchEncoder

# Train on observed traffic
codebook = AdaptiveCodebook()
for msg in training_messages:
    codebook.observe(msg)
codebook.rebuild(min_freq=5)

# Encode with learned codebook (v3 format, codebook embedded in payload)
encoder = BatchEncoder()
result = encoder.encode_batch(messages, codebook=codebook.active)
decoded = encoder.decode_batch(result.payload)  # codebook auto-extracted
```

Train a codebook from synthetic data:
```bash
python scripts/train_codebook.py --messages 50000 --benchmark
```

## Multi-Format Support

Parse and serialize 13+ formats, then compress through the batch encoder:

```python
from emergent_translator import detect_format, get_handler, BatchEncoder

fmt = detect_format("data.csv")          # "csv"
parse_fn, serialize_fn = get_handler(fmt)
records = parse_fn(open("data.csv").read())

encoder = BatchEncoder()
result = encoder.encode_batch(records)
```

Supported: JSON, CSV, JSONL, YAML, TOML, INI, XML, MessagePack, Protobuf, Parquet, Arrow, BSON, CBOR, XLSX.

## GPU Batch Encoder

For higher throughput, use the GPU-accelerated encoder (falls back to CPU with ThreadPoolExecutor if CuPy is unavailable):

```python
from emergent_translator import GPUBatchEncoder

gpu_encoder = GPUBatchEncoder(num_workers=8)
result = gpu_encoder.encode_batch(messages)
decoded = gpu_encoder.decode_batch(result.payload)
```

## LLM Token Savings

Two complementary modules for reducing token usage with LLMs like Claude:

### Code Skeletonization

Strip Python files to signatures + docstrings. Feed Claude the *structure* without paying for implementation lines:

```python
from emergent_translator import skeletonize_file

result = skeletonize_file("my_module.py", focal=["important_func"])
print(f"{result.original_tokens} -> {result.skeleton_tokens} tokens "
      f"({result.token_reduction_pct:.0f}% reduction)")
```

### Claude Text Compression

Compress keys and values in structured data flowing through Claude API conversations:

```python
from emergent_translator import ClaudeCompressor

compressor = ClaudeCompressor()
system = compressor.system_prompt_prefix() + "\n\nYour prompt..."
compressed_msgs = compressor.compress_messages(messages)
```

## Project Structure

```
src/emergent_translator/    # pip-installable package
  batch_encoder.py          # v1 batch encoder (encode/decode)
  gpu_batch_encoder.py      # v2 GPU-accelerated encoder
  adaptive_codebook.py      # v3 learned codebooks
  format_handlers.py        # 13+ format parsers
  emergent_symbols.py       # symbol encoder
  api_server.py             # FastAPI server
  cli.py                    # CLI tool
scripts/                    # benchmarks, stress tests, workers
tests/                      # 535 tests
```

## Development

```bash
git clone https://github.com/maco144/emergent-language
cd emergent-language
pip install -e ".[dev,formats]"
python -m pytest tests/ -v
```

## Docker

```bash
docker build -t emergent-translator .
docker run -p 8000:8000 emergent-translator
```

## License

GPL-3.0-or-later. See [LICENSE](LICENSE) for details.

Commercial licensing available — see [COMMERCIAL_LICENSE.md](COMMERCIAL_LICENSE.md).
