Metadata-Version: 2.4
Name: interrater-lm
Version: 0.1.0
Summary: Interrater agreement for language model evaluation in Python
Project-URL: Homepage, https://github.com/leesadie/interrater-lm
Project-URL: Documentation, https://leesadie.github.io/interrater-lm
Project-URL: Issues, https://github.com/leesadie/interrater-lm/issues
Author-email: Sadie Lee <Sadie.Lee@childrens.harvard.edu>
License-Expression: MIT
License-File: LICENSE
Keywords: interrater-agreement,llm-evaluation
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: joblib==1.5.3
Requires-Dist: krippendorff==0.8.2
Requires-Dist: narwhals==2.23.0
Requires-Dist: numpy==2.5.1
Requires-Dist: pandas==3.0.3
Requires-Dist: patsy==1.0.2
Requires-Dist: python-dateutil==2.9.0.post0
Requires-Dist: scikit-learn==1.9.0
Requires-Dist: scipy==1.18.0
Requires-Dist: six==1.17.0
Requires-Dist: statsmodels==0.14.6
Requires-Dist: threadpoolctl==3.6.0
Requires-Dist: typing-extensions==4.16.0
Description-Content-Type: text/markdown

# interrater-lm

## About

A Python package for evaluating language model-style structured outputs with agreement coefficients and statistical procedures for deriving uncertainty and significance. 

The aim of this package is not primarily to provide novel implementations of agreement coefficients but instead to make computing these simpler for comparing language models. To that end, we make use of existing libraries for some definitions, and define our own only when well-maintained implementations do not exist or significant additional preprocessing is needed to use with language model-style structured outputs.

## Documentation

The documentation is at [https://leesadie.github.io/interrater-lm](https://leesadie.github.io/interrater-lm).

## Features

- Agreement coefficients
    - Cohen's kappa - as defined by [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html)
    - Fleiss's kappa - as defined by [statsmodels](https://www.statsmodels.org/dev/generated/statsmodels.stats.inter_rater.fleiss_kappa.html)
    - Gwet's AC1 and AC2
    - Krippendorff's alpha - as defined by [fast-krippendorff](https://github.com/pln-fing-udelar/fast-krippendorff)
    - Kendall's W - derived from [scipy's Friedman test](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.friedmanchisquare.html)
    - Scott's pi
    - Percent agreement
    - Jaccard similarity
    - MASI similarity
- Data levels of measurement
    - Nominal 
    - Ordinal
    - Ratio
    - Interval
- Weighting - for Cohen's kappa and Gwet's AC2
    - Identity
    - Linear
    - Quadratic
- Statistical procedures
    - Bias-corrected and accelerated (BCa) cluster bootstrapped confidence intervals - as defined by [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html)
    - Permutation tests with optional Monte Carlo sampling for one-sided and two-sided hypotheses
- Standardized data container
- Automatic preprocessing and validation based on data level

## Installation

**GitHub**

- Main branch: https://github.com/leesadie/interrater
- Tags: https://github.com/leesadie/interrater/tags 

**PyPi**

To obtain the latest release of interrater-lm using pip:

```bash
python -m pip install interrater_lm
```

## Contributing

Contributions in any form are welcome, including:

- Documentation improvements
- Additional tests
- New features to existing coefficients, e.g., other weights
- New coefficients
- New statistical methods

See [Contributing](CONTRIBUTING.md) for guidelines.

## License

[MIT License](LICENSE.md)