Metadata-Version: 2.4
Name: hyzip
Version: 0.6.0
Summary: Auto-Routing Hybrid Text Compressor (MoeZip + TikMax)
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: moezip
Requires-Dist: tiktoken>=0.5.0

# hyzip

An auto-routing hybrid text compression engine that dynamically evaluates **libbsc** (Burrows-Wheeler Transform + LCP + ANS range coding), **MoeZip** (32-Expert MoE router with 2nd-order Markov context prediction and ANS range coding), **TikMax** (BPE tokenization with LEB128 varint packing), **LZMA2 Extreme** (7-Zip Preset 9 Extreme), **Bzip2** (Burrows-Wheeler block sorting), and **Raw Passthrough**.

It guarantees **zero data expansion** on uncompressible or micro-text inputs while maintaining high-density block compression and high throughput for larger archives, text streams, and database ingestion pipelines.

---

## Engine Roster

For every input string or byte payload, candidate backends are evaluated concurrently within their optimal workload zones. The engine selecting the absolute smallest byte count is returned prepended with a 1-byte header flag.

| Header Flag | Byte Code | Engine | Workload Scope & Architecture |
| :---: | :---: | :--- | :--- |
| `b'R'` | `0x52` | **Raw Fallback** | Uncompressed passthrough. Guarantees payload size never exceeds input length + 1 byte header. |
| `b'T'` | `0x54` | **TikMax** | BPE subword tokenization (`cl100k_base`) with LEB128 varint packing for micro inputs (1 B to 500 B). |
| `b'M'` | `0x4D` | **MoeZip** | 32-Expert MoE router + 2nd-order Markov context prediction + ANS range coding for text streams (32 B to 50 MB). |
| `b'S'` | `0x53` | **libbsc** | Primary Large Engine (BWT + LCP + ANS range coding). Delivers high-throughput block compression and sub-2.0 bpb ratios on payloads >= 512 B. |
| `b'B'` | `0x42` | **Bzip2** | Burrows-Wheeler block sorting (Level 9). Evaluated when input size >= 512 bytes. |
| `b'X'` | `0x58` | **LZMA2 Extreme** | 64 MB sliding dictionary (Preset 9 + Extreme). High-ratio fallback for payloads >= 512 bytes. |

Decompression reads the 1-byte header flag in O(1) time and routes the body directly to the corresponding decoder.

---

## Installation

```bash
pip install hyzip
```

### Automatic Binary & Compilation Management

`hyzip` manages native execution for **libbsc** automatically across platforms:
* **Windows:** Auto-downloads and caches pre-compiled native binaries on first use.
* **Linux / Termux ARM64 / macOS:** Auto-fetches the lightweight source archive and compiles a native executable using system C++ compilers (`g++`, `clang++`, `gcc`).
* **Offline Environments & Graceful Fallbacks:** If network access or C++ compilers are unavailable, `hyzip` gracefully falls back to built-in engines (`LZMA2`, `MoeZip`, `Bzip2`) without raising errors or interrupting execution.

### Optional High-Speed Dependencies

```bash
pip install tiktoken moezip
```

* If `tiktoken` is installed, `hyzip` uses C-accelerated BPE token packing.
* If optional dependencies are missing, `hyzip` falls back to pure-Python shims or skips unsupported engines gracefully.

---

## Configuration & Usage

### Basic API

```python
import hyzip

# Compress text (auto-selects absolute smallest payload)
compressed = hyzip.compress("Hello world! Do you like rock music?")

# Decompress back to string
text = hyzip.decompress(compressed)
```

### Disabling libbsc Programmatically or via Environment

`libbsc` can be toggled at runtime or configured via environment variables:

```python
import hyzip

# 1. Disable libbsc for a single compression call
compressed = hyzip.compress(data, use_bsc=False)

# 2. Disable libbsc globally across the module
hyzip.ENABLE_BSC = False
```

Alternatively, set the environment variable before launching your process:

```bash
export HYZIP_USE_BSC=0
```

### Byte Stream Input and Output

```python
import hyzip

raw_data = b"Raw binary bytes or compact JSON payload"

# Compress bytes or string
compressed = hyzip.compress(raw_data)

# Decompress and return raw bytes
decompressed_bytes = hyzip.decompress(compressed, return_bytes=True)
```

---

## Zero Payload Expansion Guarantee

Standard algorithms (`gzip`, `bzip2`, `zstd`) often inflate small payloads due to fixed dictionary header overhead (for example, compressing a 5-byte string can result in a 40-byte output).

`hyzip` includes `FLAG_RAW` (`b'R'`) in candidate evaluations:
1. If all compressed engine outputs exceed `len(input) + 1`, `hyzip` selects `FLAG_RAW`.
2. Total payload overhead is strictly capped at **input length + 1 byte**.

---

## TikMax Varint Encoding

Fixed 24-bit integer allocations waste space on small token IDs (such as token ID 68 stored as `0x000044`).

TikMax uses LEB128 variable-length integer encoding:
* Token IDs < 128: **1 byte**
* Token IDs < 16,384: **2 bytes**
* Token IDs >= 16,384: **3 bytes**

Because high-frequency tokens in BPE vocabularies map to low integer IDs, token streams are packed without requiring metadata headers.

---

## Benchmarks

Benchmark evaluation conducted on a natural language dialogue corpus containing 677,336 turns (179.35 MB original JSON dataset).

### Large Payload Performance

On multi-megabyte payloads, `libbsc` serves as the primary engine for large text blocks, reaching compression ratios below 2.0 bits per byte while scaling throughput up to nearly 70 MB/s.

| Payload Size | Raw Bytes | `hyzip` Compressed | Bits Per Byte (bpb) | Compression Speed | Throughput | Verification |
| :--- | :--- | :--- | :---: | :---: | :---: | :---: |
| **1.00 MB** | 1,048,924 B | 296,028 B | **2.258 bpb** | 0.04 s | 26.41 MB/s | **PASS (100%)** |
| **5.00 MB** | 5,244,553 B | 1,369,644 B | **2.089 bpb** | 0.10 s | 52.56 MB/s | **PASS (100%)** |
| **10.00 MB** | 10,485,777 B | 2,649,298 B | **2.021 bpb** | 0.15 s | 65.00 MB/s | **PASS (100%)** |
| **20.00 MB** | 20,972,302 B | 5,198,712 B | **1.983 bpb** | 0.29 s | **69.23 MB/s** | **PASS (100%)** |

By routing large inputs to parallel block sorting, multi-megabyte files compress in under a third of a second while achieving better compression density than traditional dictionary algorithms.

### Workload Zone Summary

| Workload Zone | Size Range | Primary Engine | Space Saved | Avg Latency | Key Advantage |
| :--- | :--- | :--- | :---: | :---: | :--- |
| **Micro Payloads** | `< 64 B` | **TikMax** (`b'T'`) | 44.57% | 0.41 ms | Zero header bloat via varint token packing |
| **Medium Payloads** | `64 B - 512 B` | **MoeZip** (`b'M'`) | 57.58% | 6.54 ms | Adaptive 32-expert Markov context prediction |
| **Large Payloads** | `> 512 B` | **libbsc** (`b'S'`) | 75.21% | 0.29 s (20MB) | Parallel BWT + LCP sorting at high throughput |

### Engine Selection Breakdown

| Engine | Header Flag | Workload Scope & Architecture |
| :--- | :---: | :--- |
| **TikMax** | `b'T'` | Micro text and ultra-short dialogue (< 64 B) |
| **MoeZip** | `b'M'` | Medium text, short paragraphs, and dialogue turns (64 B - 512 B) |
| **libbsc** | `b'S'` | **Primary Large Engine:** Multi-megabyte text blocks (> 512 B) |
| **LZMA2 Extreme** | `b'X'` | High-ratio dictionary fallback (> 512 B) |
| **Bzip2** | `b'B'` | Block sorting fallback |
| **Raw Passthrough** | `b'R'` | Uncompressible or micro strings where overhead exceeds original size |

---

## License

MIT
