Metadata-Version: 2.5
Name: zeneval
Version: 0.1.1
Summary: Create, serve, and evaluate coding benchmarks
Project-URL: Homepage, https://github.com/UYousafzai/zenith
Project-URL: Repository, https://github.com/UYousafzai/zenith
Project-URL: Issues, https://github.com/UYousafzai/zenith/issues
Author-email: Umar Yousafzai <umar.khan@deepreader.de>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: benchmarks,coding,evaluation,software-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: datasets
Requires-Dist: docker
Requires-Dist: pydantic
Requires-Dist: requests
Requires-Dist: sqlmodel
Requires-Dist: tqdm
Provides-Extra: agents
Requires-Dist: cursor-sdk; extra == 'agents'
Requires-Dist: deepagents; extra == 'agents'
Requires-Dist: langchain-anthropic; extra == 'agents'
Requires-Dist: langchain-core; extra == 'agents'
Requires-Dist: langchain-openai; extra == 'agents'
Requires-Dist: langgraph; extra == 'agents'
Provides-Extra: api
Requires-Dist: fastapi; extra == 'api'
Requires-Dist: langchain-anthropic; extra == 'api'
Requires-Dist: langchain-core; extra == 'api'
Requires-Dist: langchain-openai; extra == 'api'
Requires-Dist: langgraph; extra == 'api'
Requires-Dist: platformdirs; extra == 'api'
Requires-Dist: uvicorn[standard]; extra == 'api'
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: packaging
Requires-Dist: pyinstaller>=6.10; extra == 'packaging'
Description-Content-Type: text/markdown

# zeneval

Python library for creating, serving, and evaluating coding benchmarks.

The desktop app (`desktop/`) is the primary UI; this package exposes the core engine (sample generation, Docker sandboxes, eval harnesses, and the local API in `zeneval_api/`).

## Install

```bash
pip install -e ".[api,agents,dev]"
```

## Usage

Load SWE-bench Pro (cached under `~/.cache/zeneval`, override with `ZENEVAL_CACHE_DIR`):

```python
from zeneval.datasets import load

for sample in load("swe_bench_pro"):
    print(sample["data"]["instance_id"])
```

Convert SWE-bench Pro rows into ZenBench generator samples:

```python
from zeneval.pipelines.swe_to_zenbench import SweToZenbenchPipeline

SweToZenbenchPipeline().run(limit=10)
```

Run generation/eval via the local API (also bundled in the desktop app):

```bash
zeneval-api
```

## Package layout

| Module | Purpose |
|--------|---------|
| `zeneval.zenbench` | Sample datatypes, generation agent, eval runner |
| `zeneval_api` | FastAPI server used by the desktop app |
| `zeneval.sandbox` | Docker sandbox + container registry |
| `zeneval.harness` | Agent harness integrations (Cursor) |
| `zeneval.providers` | LLM providers for generation |
| `zeneval.pipelines` | Benchmark conversion pipelines |
| `zeneval.datasets` | Optional upstream dataset loaders |
