Metadata-Version: 2.1
Name: sotabencheval
Version: 0.0.38
Summary: Easily benchmark Machine Learning models on selected tasks and datasets
Home-page: https://sotabench.com
Author: Atlas ML
Author-email: hello@sotabench.com
Maintainer: Atlas ML
Maintainer-email: hello@sotabench.com
License: Apache-2.0
Platform: Windows
Platform: POSIX
Platform: MacOSX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
Requires-Dist: Cython
Requires-Dist: numpy
Requires-Dist: pycocotools (>=2.0.0)
Requires-Dist: sotabenchapi (>=0.0.13)
Requires-Dist: tqdm (>=4.32.2)
Requires-Dist: beautifulsoup4 (>=4.7.0)
Requires-Dist: sacrebleu (==1.4.1)

<p align="center"><img width=500 src="/docs/docs/img/sotabencheval.png"></p>

--------------------------------------------------------------------------------

[![PyPI version](https://badge.fury.io/py/sotabencheval.svg)](https://badge.fury.io/py/sotabencheval) [![Generic badge](https://img.shields.io/badge/Documentation-Here-<COLOR>.svg)](https://paperswithcode.github.io/sotabench-eval/)

`sotabencheval` is a framework-agnostic library that contains a collection of deep learning benchmarks you can use to benchmark your models. It can be used in conjunction with the [sotabench](https://www.sotabench.com) service to record results for models, so the community can compare model performance on different tasks, as well as a continuous integration style service for your repository to benchmark your models on each commit.

## Benchmarks Supported

- [ADE20K](https://paperswithcode.github.io/sotabench-eval/ade20k/) (Semantic Segmentation)
- [COCO](https://paperswithcode.github.io/sotabench-eval/coco/) (Object Detection)
- [ImageNet](https://paperswithcode.github.io/sotabench-eval/imagenet/) (Image Classification)
- [SQuAD](https://paperswithcode.github.io/sotabench-eval/squad/) (Question Answering)
- [WikiText-103](https://paperswithcode.github.io/sotabench-eval/wikitext103/) (Language Modelling)
- [WMT](https://paperswithcode.github.io/sotabench-eval/wmt/) (Machine Translation)

PRs welcome for further benchmarks! 

## Installation

Requires Python 3.6+. 

```bash
pip install sotabencheval
```

## Get Benching! 🏋️

You should read the [full documentation here](https://paperswithcode.github.io/sotabench-eval/index.html), which contains guidance on getting started and connecting to [sotabench](https://www.sotabench.com).

Integration is lightweight. For example, if you are evaluating an ImageNet model, you initialize an Evaluator object and (optionally) link to any linked paper:

```python
from sotabencheval.image_classification import ImageNetEvaluator
evaluator = ImageNetEvaluator(
             model_name='FixResNeXt-101 32x48d',
             paper_arxiv_id='1906.06423')
```

Then for each batch of predictions your model makes on ImageNet, pass a dictionary of keys as image IDs and values as a `np.ndarray`s of logits to the `evaluator.add` method:

```python
evaluator.add(output_dict=dict(zip(image_ids, batch_output)))
```

The evaluation logic just needs to be written in a `sotabench.py` file and sotabench will run it on each commit and record the results:

<a href="https://sotabench.com/user/htvr/repos/TouvronHugo/FixRes#latest-results"><img width=500 src="/docs/docs/img/results.png"></a>

## Contributing

All contributions welcome!





