Metadata-Version: 2.1
Name: rorf
Version: 0.1.1
Summary: 
Author: Tze-Yang Tung
Author-email: r0ymanesco@gmail.com
Requires-Python: >=3.10,<3.12
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: accelerate (>=0.34.2,<0.35.0)
Requires-Dist: click (>=8.1.7,<9.0.0)
Requires-Dist: datasets (>=3.0.0,<4.0.0)
Requires-Dist: einops (>=0.8.0,<0.9.0)
Requires-Dist: flash-attn (>=2.6.3,<3.0.0)
Requires-Dist: huggingface-hub (>=0.21.4,<0.22.0)
Requires-Dist: matplotlib (>=3.9.2,<4.0.0)
Requires-Dist: numpy (>=2.1.1,<3.0.0)
Requires-Dist: openai (>=1.47.1,<2.0.0)
Requires-Dist: pandas (>=2.2.3,<3.0.0)
Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
Requires-Dist: scikit-learn (>=1.5.2,<2.0.0)
Requires-Dist: tiktoken (>=0.7.0,<0.8.0)
Requires-Dist: torch (>=2.4.1,<3.0.0)
Requires-Dist: transformers (>=4.44.2,<5.0.0)
Requires-Dist: voyageai (>=0.2.3,<0.3.0)
Requires-Dist: wandb (>=0.16.4,<0.17.0)
Description-Content-Type: text/markdown

# RoRF - Routing on Random Forests

RoRF is a framework for training and serving random forest-based LLM routers.

Our core features include:
- 12 pre-trained routers across 6 model pairs and 2 embedding models ([jinaai/jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3), [voyageai/voyage-large-2-instruct](https://docs.voyageai.com/docs/embeddings#model-choices)) that reduce costs while either maintaining or improving performance.
- Our pre-trained routers outperform existing routing solutions, including open-source and commercial offerings.

## Installation
### PyPI
```sh
pip install rorf
```
### Source
```sh
git clone https://github.com/Not-Diamond/RoRF
cd RoRF
pip install -e .
```

## Quickstart
We adopt RouteLLM's [Controller](https://github.com/lm-sys/RouteLLM/tree/main?tab=readme-ov-file#quickstart) to allow users to replace their existing routing setups with RoRF. Our `Controller` requires a `router` (available either locally or on Huggingface Hub) that routes between `model_a` (usually stronger/expensive) and `model_b` (usually weaker/cheaper). Our release includes 6 model pairs between different models and providers as described in Model Support.
```python
from rorf.controller import Controller

router = Controller(
    router="notdiamond/rorf-jina-llama31405b-llama3170b",
    model_a="llama-3.1-405b-instruct",
    model_b="llama-3.1-70b-instruct",
    threshold=0.3,
)

recommended_model = router.route("What is the meaning of life?")
print(f"Recommended model: {recommended_model}")
```
We also provide a `threshold` parameter that determines the percentage of calls made to each model, allowing users to decide their own cost vs performance tradeoffs.

## Training RoRF
We include our training framework for RoRF so that users can train custom routers on their own data and model pairs. `trainer.py` is the entry-point for training, and `run_trainer.sh` provides an example command to train a model router for `llama-3.1-405b-instruct` vs `llama-3.1-70b-instruct` on top of Jina AI's embeddings.


## Motivation
Our experiments show that:
- Routing between a pair of strong and weak models can reduce costs while maintaining the strong model's performance.
- Routing between a pair of two strong models can reduce costs while outperforming both individual models.
