Metadata-Version: 2.4
Name: edgeforge
Version: 0.1.0
Summary: Developer-friendly CLI for quantizing, pruning, and exporting Hugging Face models for edge deployment.
Author: OpenAI Codex
License: MIT
Keywords: quantization,pruning,tflite,litert,gguf,llama.cpp,huggingface
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Build Tools
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: huggingface_hub>=0.30.0
Provides-Extra: gptq
Requires-Dist: optimum>=1.24.0; extra == "gptq"
Requires-Dist: auto-gptq>=0.7.1; extra == "gptq"
Provides-Extra: awq
Requires-Dist: llmcompressor>=0.5.1; extra == "awq"
Provides-Extra: bnb
Requires-Dist: bitsandbytes>=0.45.5; extra == "bnb"
Provides-Extra: litert
Requires-Dist: ai-edge-torch>=0.4.0; extra == "litert"
Requires-Dist: ai-edge-litert>=1.2.0; extra == "litert"
Provides-Extra: tflite
Requires-Dist: tensorflow>=2.19.0; extra == "tflite"
Requires-Dist: onnx>=1.18.0; extra == "tflite"
Requires-Dist: onnxruntime>=1.21.0; extra == "tflite"
Requires-Dist: onnx2tf>=1.28.2; extra == "tflite"
Provides-Extra: gguf
Requires-Dist: llama-cpp-python>=0.3.8; extra == "gguf"
Requires-Dist: gguf>=0.10.0; extra == "gguf"
Requires-Dist: auto-round>=0.5.0; extra == "gguf"
Provides-Extra: datasets
Requires-Dist: datasets>=3.5.0; extra == "datasets"
Provides-Extra: torch
Requires-Dist: transformers>=4.51.0; extra == "torch"
Requires-Dist: accelerate>=1.6.0; extra == "torch"
Requires-Dist: torch>=2.6.0; extra == "torch"
Provides-Extra: dev
Requires-Dist: pytest>=8.3.0; extra == "dev"
Provides-Extra: all
Requires-Dist: optimum>=1.24.0; extra == "all"
Requires-Dist: auto-gptq>=0.7.1; extra == "all"
Requires-Dist: llmcompressor>=0.5.1; extra == "all"
Requires-Dist: bitsandbytes>=0.45.5; extra == "all"
Requires-Dist: ai-edge-torch>=0.4.0; extra == "all"
Requires-Dist: ai-edge-litert>=1.2.0; extra == "all"
Requires-Dist: tensorflow>=2.19.0; extra == "all"
Requires-Dist: onnx>=1.18.0; extra == "all"
Requires-Dist: onnxruntime>=1.21.0; extra == "all"
Requires-Dist: onnx2tf>=1.28.2; extra == "all"
Requires-Dist: llama-cpp-python>=0.3.8; extra == "all"
Requires-Dist: gguf>=0.10.0; extra == "all"
Requires-Dist: auto-round>=0.5.0; extra == "all"
Requires-Dist: datasets>=3.5.0; extra == "all"
Requires-Dist: transformers>=4.51.0; extra == "all"
Requires-Dist: accelerate>=1.6.0; extra == "all"
Requires-Dist: torch>=2.6.0; extra == "all"
Requires-Dist: pytest>=8.3.0; extra == "all"

# EdgeForge

**Quantize, prune, and deploy Hugging Face LLMs to Google AI Edge Gallery (LiteRT / TFLite) from your terminal.**

EdgeForge is a developer-friendly CLI toolkit that downloads Hugging Face transformer models, prepares them for compression, applies quantization and pruning workflows, and stages export artifacts for:

- LiteRT `.task` for Google AI Edge Gallery
- TFLite `.tflite` for mobile inference
- GGUF `.gguf` workflows for `llama.cpp`

## Install

```bash
pip install edgeforge
pip install "edgeforge[gptq]"
pip install "edgeforge[awq]"
pip install "edgeforge[litert,tflite]"
pip install "edgeforge[gguf]"
pip install "edgeforge[all]"
```

For this workspace:

```powershell
.\enve\python.exe -m pip install -e .
```

## CLI overview

```bash
edgeforge auth login
edgeforge auth status
edgeforge download google/gemma-2b-it
edgeforge quantize google/gemma-2b-it --method gptq --bits 4
edgeforge prune ./models/gemma-2b-it-gptq --method magnitude --sparsity 0.3
edgeforge convert ./models/gemma-2b-it-gptq-pruned --format litert
edgeforge run google/gemma-2b-it --quant-method awq --bits 4 --export-format gguf
edgeforge chat ./models/gemma-2b-it.gguf
```

## Step By Step For A New Model

Use this workflow when you want to process a new Hugging Face model from scratch.

### 1. Activate the environment

```cmd
C:\Quanitization\enve\Scripts\activate
cd C:\Quanitization
```

### 2. Authenticate with Hugging Face

Only needed for gated or private models.

```cmd
.\enve\Scripts\edgeforge.exe auth login --token hf_xxx
```

### 3. Download the model

Public model example:

```cmd
.\enve\Scripts\edgeforge.exe download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --no-auth
```

Gated model example:

```cmd
.\enve\Scripts\edgeforge.exe download google/gemma-2b-it
```

### 4. Quantize the model

For the most reliable GGUF workflow, use `fp16` first.

```cmd
.\enve\Scripts\edgeforge.exe quantize "C:\Users\vicky\.edgeforge\models\MODEL_FOLDER_NAME" --method fp16
```

Example:

```cmd
.\enve\Scripts\edgeforge.exe quantize "C:\Users\vicky\.edgeforge\models\TinyLlama--TinyLlama-1.1B-Chat-v1.0" --method fp16
```

### 5. Optional pruning

```cmd
.\enve\Scripts\edgeforge.exe prune "C:\Users\vicky\.edgeforge\artifacts\MODEL_NAME-fp16-16bit" --method magnitude --sparsity 0.1
```

Example:

```cmd
.\enve\Scripts\edgeforge.exe prune "C:\Users\vicky\.edgeforge\artifacts\TinyLlama--TinyLlama-1.1B-Chat-v1.0-fp16-16bit" --method magnitude --sparsity 0.1
```

### 6. Export to GGUF

Without pruning:

```cmd
.\enve\Scripts\edgeforge.exe convert "C:\Users\vicky\.edgeforge\artifacts\MODEL_NAME-fp16-16bit" --format gguf
```

With pruning:

```cmd
.\enve\Scripts\edgeforge.exe convert "C:\Users\vicky\.edgeforge\artifacts\MODEL_NAME-fp16-16bit-pruned-magnitude-10" --format gguf
```

### 7. Find the GGUF file

```cmd
dir "C:\Users\vicky\.edgeforge\exports\MODEL_EXPORT_FOLDER\gguf" /s
```

### 8. Run chat with llama.cpp

```cmd
.\enve\Scripts\edgeforge.exe chat "FULL_PATH_TO_MODEL.gguf" --backend gguf --executable "C:\Users\vicky\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\llama-cli.exe"
```

### Recommended order

1. Download the model.
2. Quantize with `fp16`.
3. Skip pruning for the first test.
4. Export to `gguf`.
5. Test chat with `llama.cpp`.
6. Add pruning only after the base export works.

### Full example

```cmd
.\enve\Scripts\edgeforge.exe download microsoft/phi-2 --no-auth
.\enve\Scripts\edgeforge.exe quantize "C:\Users\vicky\.edgeforge\models\microsoft--phi-2" --method fp16
.\enve\Scripts\edgeforge.exe convert "C:\Users\vicky\.edgeforge\artifacts\microsoft--phi-2-fp16-16bit" --format gguf
dir "C:\Users\vicky\.edgeforge\exports\microsoft--phi-2-fp16-16bit\gguf" /s
```

## INT8 Workflow

Use this path when you want an INT8-compressed Hugging Face artifact first.

### 1. Quantize to INT8

```cmd
.\enve\Scripts\edgeforge.exe quantize "C:\Users\vicky\.edgeforge\models\MODEL_FOLDER_NAME" --method int8
```

Example:

```cmd
.\enve\Scripts\edgeforge.exe quantize "C:\Users\vicky\.edgeforge\models\TinyLlama--TinyLlama-1.1B-Chat-v1.0" --method int8
```

### 2. Optional pruning

```cmd
.\enve\Scripts\edgeforge.exe prune "C:\Users\vicky\.edgeforge\artifacts\MODEL_NAME-int8-8bit" --method magnitude --sparsity 0.1
```

### 3. Important note for GGUF export

- INT8 artifacts are useful for local Hugging Face style workflows.
- For GGUF export, `fp16` is usually the safer source format.
- If GGUF conversion from INT8 fails, convert from the `fp16` artifact instead.

## LiteRT Workflow

Use this path only when you have real LiteRT/TFLite conversion dependencies installed.

### 1. Install conversion backends

```cmd
.\enve\python.exe -m pip install ai-edge-torch tensorflow
```

### 2. Export to LiteRT

```cmd
.\enve\Scripts\edgeforge.exe convert "C:\Users\vicky\.edgeforge\artifacts\MODEL_NAME" --format litert
```

### 3. Verify the output

- A real LiteRT `.task` file should contain a non-trivial `model.tflite`.
- If the bundle contains a tiny placeholder `model.tflite`, the real backend was not used.
- For many general Hugging Face LLMs, LiteRT conversion is still model-dependent and not guaranteed.

### 4. Best candidates

- Gemma-family models
- Phi-family models
- Smaller transformer models with simpler operator coverage

## Common Errors And Fixes

### Gated repo error

Error:

```text
GatedRepoError / 403 Client Error
```

Fix:

- Request access on Hugging Face.
- Or test with a public model first.

### `bitsandbytes` not installed

Error:

```text
bitsandbytes is not installed or not supported
```

Fix:

```cmd
.\enve\python.exe -m pip install bitsandbytes
```

If that still fails on Windows, use `fp16` instead.

### `sentencepiece` missing during GGUF export

Error:

```text
ModuleNotFoundError: No module named 'sentencepiece'
```

Fix:

```cmd
.\enve\python.exe -m pip install sentencepiece
```

### GGUF folder exists but no `.gguf` file

Fix:

- Remove the broken export folder.
- Rerun conversion with the latest EdgeForge code.
- Prefer `fp16` as the input artifact for GGUF export.

### LiteRT `.task` exists but is not real

Signs:

- `model.tflite` inside the bundle is only a few hundred bytes
- `ai-edge-torch` and `tensorflow` are not installed

Fix:

```cmd
.\enve\python.exe -m pip install ai-edge-torch tensorflow
```

Then rerun:

```cmd
.\enve\Scripts\edgeforge.exe convert "C:\Users\vicky\.edgeforge\artifacts\MODEL_NAME" --format litert
```

### `llama-cli.exe` not found

Error:

```text
FileNotFoundError: [WinError 2]
```

Fix:

- Use the real path to `llama-cli.exe`
- Do not leave `C:\path\to\llama-cli.exe` as a placeholder

## Current backend status

- `fp16` quantization is implemented with real Hugging Face model loading and saving.
- `dynamic` quantization uses real PyTorch dynamic quantization for `Linear` layers.
- `gptq` and `awq` are wired through `llmcompressor.oneshot()` with generated recipes.
- `gguf` export auto-detects `auto-round` plus `gguf` and uses the Python export path when available.
- `tflite` and `litert` use a real `ai-edge-torch` conversion path when that package is installed; otherwise EdgeForge writes an explicit fallback artifact instead of pretending conversion succeeded.

## Package layout

```text
src/edgeforge/
  __init__.py
  __main__.py
  auth.py
  chat.py
  cli.py
  config.py
  converter.py
  downloader.py
  pipeline.py
  pruner.py
  quantizer.py
  utils.py
SKILL.md
tests/
```

## Reality check for LiteRT

EdgeForge is intentionally honest about deployment constraints:

- LiteRT export is best for models and operator sets that are already compatible with Google AI Edge tooling.
- General Hugging Face decoder-only LLMs often need model-family-specific conversion work before they become valid `.task` or `.tflite` artifacts.
- GGUF plus `llama.cpp` remains the most practical local runtime for many larger LLMs.

## AWQ note

- `AutoAWQ` is deprecated upstream.
- EdgeForge now treats `llmcompressor` as the recommended AWQ dependency path.
- `llmcompressor` is documented upstream with Linux as the recommended environment for GPU workflows.
- On Windows, very large AWQ workflows may still be less reliable than Linux GPU environments.

The current project provides a strong orchestration layer, clear manifests, conversion plans, and extension points instead of pretending every Hugging Face model can be converted to LiteRT in one generic step.

## Development

```bash
pip install -e ".[dev]"
pytest tests/ -v
python -m build
```
