Metadata-Version: 2.1
Name: maibenchai
Version: 0.2
Summary: A simple package for LLM application evaluation.
Author: maibench
Author-email: maibench <support@maibench.ai>
License: MIT License
        
        Copyright (c) 2023 maibench
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Project-URL: Homepage, https://maibench.ai
Requires-Python: >=3.7.1
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests >=2.31.0

### maibench Python SDK

Python toolings to help run evaluations on large langauge model chains.

[![](https://img.shields.io/badge/Visit%20Us!-maibench.ai-brightgreen)](https://maibench.ai)

 ## Getting started
 First, install the package.

 `pip install maibenchai`

Next, create a completion function. This maps different inputs your LLM application response. Here's an example.

```
 from maibench import CompletionFn

 class ExampleCompletionFn(CompletionFn):
    @staticmethod
    def complete(**kwargs) -> str:
        return "This is an example response from an LLM application."
```

Next, create some text cases. These test cases should each contain the input kwargs of your completion function in additon to an expected output and a grade_fn. Supported grade functions are currently `match`, `includes`, `fuzzy_match`, `not_match`, `not_includes`, and `not_fuzzy_match`. More complex grading functions and model-based grading is coming soon.

``` 
EXAMPLE_TEST_CASES = [
    {
        "input": "This is an example input.",
        "expected": "This is an example response from an LLM application.",
        "grade_fn": "match",
    },
    {
        "input": "This is an example input 2.",
        "expected": "This is an example response from an LLM application.",
        "grade_fn": "match",
    },
]
 ```

 Create and run.

 ```
EXAMPLE_TEST_SUITE = TestSuite(name="maibench.example_test_suite", completion_fn=ExampleCompletionFn, test_cases=EXAMPLE_TEST_CASES)
results, id = EXAMPLE_TEST_SUITE.run()
```

See results at https://maibench.ai/individual-result?id=[INSERT ID]

### We hope you enjoy!

Contact support@maibench.ai with any questions.
