Metadata-Version: 2.1
Name: turkish-lm-tuner
Version: 0.1.4
Summary: Implementation of the Turkish LM Tuner
Author-email: Gökçe Uludoğan <gokceuludogan@gmail.com>, Zeynep Yirmibeşoğlu Balal <yirmibesogluz@gmail.com>, Furkan Akkurt <furkanakkurt9285@gmail.com>
License: Apache-2.0
Project-URL: Source Code, https://github.com/boun-tabi-LMG/turkish-lm-tuner
Project-URL: Bug Tracker, https://github.com/boun-tabi-LMG/turkish-lm-tuner/issues
Project-URL: Documentation, https://turkish-lm-tuner-docs.boun-tabi-LMG.github.io/
Keywords: nlp,turkish,language models,finetuning
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Natural Language :: Turkish
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tqdm
Requires-Dist: hydra-core
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scikit-learn
Requires-Dist: torch >=2.0
Requires-Dist: transformers >=4.35
Requires-Dist: datasets
Requires-Dist: tokenizers
Requires-Dist: sentencepiece
Requires-Dist: accelerate
Requires-Dist: safetensors
Requires-Dist: evaluate
Requires-Dist: wandb
Requires-Dist: huggingface-hub
Requires-Dist: absl-py
Requires-Dist: rouge-score
Requires-Dist: sacrebleu
Requires-Dist: seqeval

<h1 align="center">  🦖 Turkish LM Tuner </h1>
<!--<h4 align="center"> Summary of project or library comes here. </h4>-->

</br>

[![Paper](https://img.shields.io/badge/DOI-10.18653/v1/2024.findings--acl.600-blue)](https://aclanthology.org/2024.findings-acl.600/)
[![Code license](https://img.shields.io/badge/Code%20License-MIT-green.svg)](https://github.com/boun-tabi-LMG/blob/main/LICENSE)
[![PyPI](https://img.shields.io/pypi/v/turkish-lm-tuner)](https://pypi.org/project/turkish-lm-tuner/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/turkish-lm-tuner)](https://pypi.org/project/turkish-lm-tuner/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/turkish-lm-tuner)](https://pypi.org/project/turkish-lm-tuner/)
[![GitHub Repo stars](https://img.shields.io/github/stars/boun-tabi-LMG/turkish-lm-tuner)](https://github.com/boun-tabi-LMG/turkish-lm-tuner/stargazers)

## Overview

Turkish LM Tuner is a library for fine-tuning Turkish language models on various NLP tasks. It is built on top of [Hugging Face Transformers](https://github.com/huggingface/transformers) library. It supports finetuning with conditional generation and sequence classification tasks. The library is designed to be modular and extensible. It is easy to add new tasks and models. The library also provides data loaders for various Turkish NLP datasets.

## Installation

You can install `turkish-lm-tuner` via PyPI: 

```bash

pip install turkish-lm-tuner
```

Alternatively, you can use the following command to install the library:

```bash

pip install git+https://github.com/boun-tabi-LMG/turkish-lm-tuner.git
```

## Model Support

Any Encoder or ConditionalGeneration model that is compatible with Hugging Face Transformers library can be used with Turkish LM Tuner. The following models are tested and supported.

- [TURNA](https://arxiv.org/abs/2401.14373)
- [mT5](https://aclanthology.org/2021.naacl-main.41/)
- [mBART](https://aclanthology.org/2020.tacl-1.47/)
- [BERTurk](https://github.com/stefan-it/turkish-bert)

## Task and Dataset Support

| Task                           | Datasets                                                                                                 |
| ------------------------------ | --------------------------------------------------------------------------------------------------------                                                                                                             |
| Text Classification            | [Product Reviews](https://huggingface.co/datasets/turkish_product_reviews), [TTC4900](https://dx.doi.org/10.5505/pajes.2018.15931), [Tweet Sentiment](https://ieeexplore.ieee.org/document/8554037)                  |                                                                                                                                 |
| Natural Language Inference     | [NLI_TR](https://aclanthology.org/2020.emnlp-main.662/), [SNLI_TR](https://aclanthology.org/2020.emnlp-main.662/), [MultiNLI_TR](https://aclanthology.org/2020.emnlp-main.662/)                                      |
| Semantic Textual Similarity    | [STSb_TR](https://aclanthology.org/2021.gem-1.3/)                                                                                     |
| Named Entity Recognition       | [WikiANN](https://aclanthology.org/P19-1015/), [Milliyet NER](https://doi.org/10.1017/S135132490200284X)                                                          |
| Part-of-Speech Tagging         | [BOUN](https://universaldependencies.org/treebanks/tr_boun/index.html), [IMST](https://universaldependencies.org/treebanks/tr_imst/index.html)                                                                     |
| Text Summarization             | [TR News](https://doi.org/10.1007/s10579-021-09568-y), [MLSUM](https://aclanthology.org/2020.emnlp-main.647/), [Combined TR News and MLSUM](https://doi.org/10.1017/S1351324922000195)                        |
| Title Generation               | [TR News](https://doi.org/10.1007/s10579-021-09568-y), [MLSUM](https://aclanthology.org/2020.emnlp-main.647/), [Combined TR News and MLSUM](https://doi.org/10.1017/S1351324922000195)                        |
| Paraphrase Generation          | [OpenSubtitles](https://aclanthology.org/2022.icnlsp-1.14/), [Tatoeba](https://aclanthology.org/2022.icnlsp-1.14/), [TED Talks](https://aclanthology.org/2022.icnlsp-1.14/)                                 |


## Usage
The tutorials in the [documentation](docs/) can help you get started with `turkish-lm-tuner`.

## Examples

### Fine-tune and evaluate a conditional generation model

```python
from turkish_lm_tuner import DatasetProcessor, TrainerForConditionalGeneration

dataset_name = "tr_news"
task = "summarization"
task_format="conditional_generation"
model_name = "boun-tabi-LMG/TURNA"
max_input_length = 764
max_target_length = 128
dataset_processor = DatasetProcessor(
    dataset_name=dataset_name, task=task, task_format=task_format, task_mode='',
    tokenizer_name=model_name, max_input_length=max_input_length, max_target_length=max_target_length
)

train_dataset = dataset_processor.load_and_preprocess_data(split='train')
eval_dataset = dataset_processor.load_and_preprocess_data(split='validation')
test_dataset = dataset_processor.load_and_preprocess_data(split="test")

training_params = {
    'num_train_epochs': 10,
    'per_device_train_batch_size': 4,
    'per_device_eval_batch_size': 4,
    'output_dir': './', 
    'evaluation_strategy': 'epoch',
    'save_strategy': 'epoch',
    'predict_with_generate': True    
}
optimizer_params = {
    'optimizer_type': 'adafactor',
    'scheduler': False,
}

model_trainer = TrainerForConditionalGeneration(
    model_name=model_name, task=task,
    optimizer_params=optimizer_params,
    training_params=training_params,
    model_save_path="turna_summarization_tr_news",
    max_input_length=max_input_length,
    max_target_length=max_target_length, 
    postprocess_fn=dataset_processor.dataset.postprocess_data
)

trainer, model = model_trainer.train_and_evaluate(train_dataset, eval_dataset, test_dataset)

model.save_pretrained(model_save_path)
dataset_processor.tokenizer.save_pretrained(model_save_path)
```

### Evaluate a conditional generation model with custom generation config

```python
from turkish_lm_tuner import DatasetProcessor, EvaluatorForConditionalGeneration

dataset_name = "tr_news"
task = "summarization"
task_format="conditional_generation"
model_name = "boun-tabi-LMG/TURNA"
task_mode = ''
max_input_length = 764
max_target_length = 128
dataset_processor = DatasetProcessor(
    dataset_name, task, task_format, task_mode,
    model_name, max_input_length, max_target_length
)

test_dataset = dataset_processor.load_and_preprocess_data(split="test")

test_params = {
    'per_device_eval_batch_size': 4,
    'output_dir': './',
    'predict_with_generate': True
}

model_path = "turna_tr_news_summarization"
generation_params = {
    'num_beams': 4,
    'length_penalty': 2.0,
    'no_repeat_ngram_size': 3,
    'early_stopping': True,
    'max_length': 128,
    'min_length': 30,
}
evaluator = EvaluatorForConditionalGeneration(
    model_path, model_name, task, max_input_length, max_target_length, test_params,
    generation_params, dataset_processor.dataset.postprocess_data
)
results = evaluator.evaluate_model(test_dataset)
print(results)
```

## Reference

If you use this repository, please cite the following related [paper](https://aclanthology.org/2024.findings-acl.600/):

```bibtex
@inproceedings{uludogan-etal-2024-turna,
    title = "{TURNA}: A {T}urkish Encoder-Decoder Language Model for Enhanced Understanding and Generation",
    author = {Uludo{\u{g}}an, G{\"o}k{\c{c}}e  and
      Balal, Zeynep  and
      Akkurt, Furkan  and
      Turker, Meliksah  and
      Gungor, Onur  and
      {\"U}sk{\"u}darl{\i}, Susan},
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-acl.600",
    doi = "10.18653/v1/2024.findings-acl.600",
    pages = "10103--10117",
}
```

## License

Note that all datasets belong to their respective owners. If you use the datasets provided by this library, please cite the original source.

This code base is licensed under the MIT license. See [LICENSE](license.md) for details.
