Metadata-Version: 2.1
Name: translation-canvas
Version: 1.0.2
Summary: Translation Canvas - A tool for evaluating and visualizing machine translation models
Author-email: Chinmay Dandekar <cdandekar@ucsb.edu>
Project-URL: Homepage, https://github.com/ChinDandekar/instructscore_visualizer
Project-URL: Issues, https://github.com/ChinDandekar/instructscore_visualizer/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12.2
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: absl-py ==2.1.0
Requires-Dist: accelerate ==0.29.1
Requires-Dist: aiohttp ==3.9.3
Requires-Dist: aiosignal ==1.3.1
Requires-Dist: annotated-types ==0.6.0
Requires-Dist: appdirs ==1.4.4
Requires-Dist: astunparse ==1.6.3
Requires-Dist: attrs ==23.2.0
Requires-Dist: blessed ==1.20.0
Requires-Dist: blinker ==1.7.0
Requires-Dist: build ==1.2.1
Requires-Dist: certifi ==2024.2.2
Requires-Dist: cffi ==1.16.0
Requires-Dist: charset-normalizer ==3.3.2
Requires-Dist: click ==8.1.7
Requires-Dist: cmake ==3.30.0
Requires-Dist: contourpy ==1.2.1
Requires-Dist: cryptography ==43.0.0
Requires-Dist: cycler ==0.12.1
Requires-Dist: cython ==0.29.36
Requires-Dist: datasets ==2.18.0
Requires-Dist: deepspeed ==0.14.0
Requires-Dist: dill ==0.3.8
Requires-Dist: docker-pycreds ==0.4.0
Requires-Dist: docstring-parser ==0.16
Requires-Dist: docutils ==0.21.2
Requires-Dist: duckdb ==1.0.0
Requires-Dist: fasteners ==0.19
Requires-Dist: filelock ==3.13.3
Requires-Dist: fire ==0.6.0
Requires-Dist: flask ==3.0.3
Requires-Dist: flask-cors ==4.0.0
Requires-Dist: flatbuffers ==24.3.25
Requires-Dist: fonttools ==4.51.0
Requires-Dist: frozenlist ==1.4.1
Requires-Dist: fsspec ==2024.2.0
Requires-Dist: gast ==0.5.4
Requires-Dist: gitdb ==4.0.11
Requires-Dist: gitpython ==3.1.43
Requires-Dist: google-pasta ==0.2.0
Requires-Dist: gpustat ==1.1.1
Requires-Dist: grpcio ==1.62.1
Requires-Dist: h5py ==3.10.0
Requires-Dist: hjson ==3.1.0
Requires-Dist: huggingface-hub ==0.23.4
Requires-Dist: idna ==3.6
Requires-Dist: inquirerpy ==0.3.4
Requires-Dist: itsdangerous ==2.1.2
Requires-Dist: jaraco-classes ==3.4.0
Requires-Dist: jaraco-context ==5.3.0
Requires-Dist: jaraco-functools ==4.0.1
Requires-Dist: jeepney ==0.8.0
Requires-Dist: jinja2 ==3.1.3
Requires-Dist: joblib ==1.4.0
Requires-Dist: jsonschema ==4.21.1
Requires-Dist: jsonschema-specifications ==2023.12.1
Requires-Dist: keras ==3.1.1
Requires-Dist: keyring ==25.2.1
Requires-Dist: kiwisolver ==1.4.5
Requires-Dist: libclang ==18.1.1
Requires-Dist: markdown ==3.6
Requires-Dist: markdown-it-py ==3.0.0
Requires-Dist: markupsafe ==2.1.5
Requires-Dist: matplotlib ==3.8.4
Requires-Dist: mdurl ==0.1.2
Requires-Dist: ml-dtypes ==0.3.2
Requires-Dist: more-itertools ==10.3.0
Requires-Dist: mpmath ==1.3.0
Requires-Dist: multidict ==6.0.5
Requires-Dist: multiprocess ==0.70.16
Requires-Dist: namex ==0.0.7
Requires-Dist: networkx ==3.3
Requires-Dist: nh3 ==0.2.18
Requires-Dist: ninja ==1.11.1.1
Requires-Dist: numpy ==1.26.4
Requires-Dist: opt-einsum ==3.3.0
Requires-Dist: optree ==0.11.0
Requires-Dist: pandas ==2.2.1
Requires-Dist: peft ==0.10.0
Requires-Dist: pfzy ==0.3.4
Requires-Dist: pillow ==10.3.0
Requires-Dist: pkginfo ==1.10.0
Requires-Dist: plotly ==5.23.0
Requires-Dist: portalocker ==2.10.1
Requires-Dist: protobuf ==4.25.3
Requires-Dist: psutil ==5.9.8
Requires-Dist: py-cpuinfo ==9.0.0
Requires-Dist: pyarrow ==15.0.2
Requires-Dist: pyarrow-hotfix ==0.6
Requires-Dist: pycparser ==2.22
Requires-Dist: pydantic ==2.6.1
Requires-Dist: pydantic-core ==2.16.2
Requires-Dist: pynvml ==11.5.0
Requires-Dist: pyparsing ==3.1.2
Requires-Dist: pyproject-hooks ==1.1.0
Requires-Dist: python-dateutil ==2.9.0.post0
Requires-Dist: python-dotenv ==1.0.1
Requires-Dist: pytz ==2024.1
Requires-Dist: pyyaml ==6.0.1
Requires-Dist: readme-renderer ==44.0
Requires-Dist: referencing ==0.35.1
Requires-Dist: regex ==2023.12.25
Requires-Dist: requests ==2.31.0
Requires-Dist: requests-toolbelt ==1.0.0
Requires-Dist: rfc3986 ==2.0.0
Requires-Dist: rich ==13.7.1
Requires-Dist: rocher ==0.50.0
Requires-Dist: rpds-py ==0.18.1
Requires-Dist: sacrebleu ==2.4.2
Requires-Dist: safetensors ==0.4.2
Requires-Dist: scikit-learn ==1.4.1.post1
Requires-Dist: scipy ==1.13.0
Requires-Dist: secretstorage ==3.3.3
Requires-Dist: sentencepiece
Requires-Dist: sentry-sdk ==1.44.1
Requires-Dist: setproctitle ==1.3.3
Requires-Dist: shtab ==1.7.1
Requires-Dist: smmap ==5.0.1
Requires-Dist: sympy ==1.12
Requires-Dist: tensorboard ==2.16.2
Requires-Dist: tensorboard-data-server ==0.7.2
Requires-Dist: tensorflow ==2.16.1
Requires-Dist: termcolor ==2.4.0
Requires-Dist: threadpoolctl ==3.4.0
Requires-Dist: tokenizers ==0.15.2
Requires-Dist: toml ==0.10.2
Requires-Dist: torch ==2.2.2
Requires-Dist: tqdm ==4.66.2
Requires-Dist: transformers ==4.39.3
Requires-Dist: trl ==0.8.1
Requires-Dist: twine ==5.1.1
Requires-Dist: tyro ==0.8.2
Requires-Dist: tzdata ==2024.1
Requires-Dist: unbabel-comet ==2.2.2
Requires-Dist: urllib3 ==2.2.1
Requires-Dist: wandb ==0.16.6
Requires-Dist: werkzeug ==3.0.2
Requires-Dist: wrapt ==1.16.0
Requires-Dist: xxhash ==3.4.1
Requires-Dist: yarl ==1.9.4

# Translation Canvas

[![PyPI version](https://badge.fury.io/py/translation-canvas.svg)](https://badge.fury.io/py/translation-canvas)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Overview

**Translation Canvas** is a Python package designed for in-depth analysis and visualization of machine translation (MT) model outputs. It facilitates both system-level and instance-level evaluations, helping researchers identify, analyze, and comprehend the strengths and weaknesses of translation models.

The tool integrates multiple evaluation metrics, including BLEU, COMET, and InstructScore, to provide a comprehensive view of translation quality. Moreover, it offers detailed natural language explanations for identified errors, powered by InstructScore, and presents the results in an intuitive and interactive dashboard.

### Why Translation Canvas?

With the rapid development of machine translation systems, traditional evaluation tools like COMET and SacreBLEU often fall short in providing fine-grained insights. **Translation Canvas** bridges this gap by offering:

**Instance-level Error Analysis:** Highlight specific errors in translation instances and explain their nature using natural language descriptions.

**System-level Insights:** Aggregate error analysis to identify common pitfalls and strengths across entire datasets.

**Visual Comparisons:** Interactive dashboard for comparing the performance of different models on a granular level.

## Installation

You can easily install **Translation Canvas** via `pip`:

```bash
pip install translation-canvas
```

After installation, run a one-time setup script to configure necessary dependencies:

```bash
translation-canvas-setup
```

For the latest development version, you can install directly from the GitHub repository:

```bash
pip install git+https://github.com/ChinDandekar/translation_canvas
```

## Features 
<div style="text-align: center;">
  <img src="https://raw.githubusercontent.com/ChinDandekar/translation_canvas/main/images/instance-3-compare.png" alt="Compare Instances" width="1000"/>
</div>

### Error Analysis

**Translation Canvas** highlights errors at the instance level using color-coded spans. Hovering over these spans reveals natural language explanations, making it easier to understand the type of errors encountered.

### Comparison and Search

**Translation Canvas** allows users to compare multiple models simulatneously. It also provides a powerful search feature to users, allowing them to filter instancs by text, error type, error scale and error explanation


### System-level Dashboard

<div style="text-align: center;">
  <img src="https://raw.githubusercontent.com/ChinDandekar/translation_canvas/main/images/system-3-compare.png" alt="Compare Systems" width="1000"/>
</div>

**Translation Canvas** provides a system-level dashboard to understand model performance at a system level.

### Instance Submission

<div style="text-align: center;">
  <img src="https://raw.githubusercontent.com/ChinDandekar/translation_canvas/main/images/submit-workflow.png" alt="Submit Workflow" width="1000"/>
</div>

Submit source-prediction-reference triplets through the user interface or by directly uploading files. The tool will process these instances, evaluate them using the integrated metrics, and provide detailed feedback.

## Usage

**Translation Canvas** operates as a web application, running in your browser. To start the application:

```bash
translation-canvas-start
```

By default, the app will be available at 
[http://127.0.0.1:5000](http://127.0.0.1:5000).

To run the app on a different port:

```bash
translation-canvas-start --port your-port
```


### Port Forwarding

If you are running **Translation Canvas** on a remote server via SSH, use port forwarding to access the app:

```bash
ssh -L your-port:127.0.0.1:5000 username@yourserver.com
```


## Evaluation and Feedback

**Translation Canvas** has been tested with machine translation experts, who found it to be both effective and user-friendly. The tool has shown to be particularly useful in pinpointing subtle errors that might be overlooked by traditional evaluation methods.



