Metadata-Version: 2.1
Name: unimernet
Version: 0.2.1
Summary: UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition
Home-page: https://github.com/opendatalab/UniMERNet
License: Apache-2.0
Keywords: MER,latex,markdown,pdf
Author: Bin Wang
Author-email: ictwangbin@gmail.com
Requires-Python: >=3.10
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Provides-Extra: full
Requires-Dist: albumentations (>=1.4.4,<2.0.0)
Requires-Dist: eva-decord (>=0.6.1,<0.7.0)
Requires-Dist: evaluate (>=0.4.1,<0.5.0)
Requires-Dist: fairscale (>=0.4.13,<0.5.0)
Requires-Dist: ftfy (>=6.2.0,<7.0.0) ; python_version >= "3.10" and python_version < "4.0"
Requires-Dist: iopath (>=0.1.9,<0.2.0)
Requires-Dist: jupyterlab (>=4.1.6,<5.0.0) ; extra == "full"
Requires-Dist: matplotlib (>=3.8.4,<4.0.0)
Requires-Dist: nltk (>=3.8.1,<4.0.0) ; extra == "full"
Requires-Dist: omegaconf (>=2.3.0,<3.0.0)
Requires-Dist: opencv-python (>=4.6.0,<5.0.0)
Requires-Dist: pandas (>=2.2.2,<3.0.0) ; extra == "full"
Requires-Dist: pdf2image (>=1.17.0,<2.0.0) ; extra == "full"
Requires-Dist: pypdfium2 (>=4.29.0,<5.0.0) ; extra == "full"
Requires-Dist: rapidfuzz (>=3.8.1,<4.0.0)
Requires-Dist: rich (>=13.7.1,<14.0.0) ; extra == "full"
Requires-Dist: streamlit (>=1.33.0,<2.0.0) ; extra == "full"
Requires-Dist: streamlit_drawable_canvas (>=0.9.3,<0.10.0) ; extra == "full"
Requires-Dist: tabulate (>=0.9.0,<0.10.0) ; extra == "full"
Requires-Dist: termcolor (>=2.4.0,<3.0.0) ; extra == "full"
Requires-Dist: timm (>=0.9.16,<0.10.0)
Requires-Dist: torch (>=2.2.2,<=2.3.1)
Requires-Dist: torchtext (>=0.17.2,<=0.18.0)
Requires-Dist: torchvision (>=0.17.2,<=0.18.1)
Requires-Dist: transformers (==4.42.4)
Requires-Dist: wand (>=0.6.13,<0.7.0)
Requires-Dist: webdataset (>=0.2.86,<0.3.0)
Project-URL: Repository, https://github.com/opendatalab/UniMERNet
Description-Content-Type: text/markdown




<div align="center">

English | [简体中文](./README-zh_CN.md)


<h1>UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition</h1>


[[ Paper ]](https://arxiv.org/abs/2404.15254) [[ Website ]](https://github.com/opendatalab/UniMERNet/tree/main) [[ Dataset (OpenDataLab)]](https://opendatalab.com/OpenDataLab/UniMER-Dataset) [[ Dataset (Hugging Face) ]](https://huggingface.co/datasets/wanderkid/UniMER_Dataset)


[[Models 🤗(Hugging Face)]](https://huggingface.co/wanderkid/unimernet_base)
[[Models <img src="./asset/images/modelscope_logo.png" width="20px">(ModelScope)]](https://www.modelscope.cn/models/wanderkid/unimernet_base)

🔥🔥 [CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation](https://github.com/opendatalab/UniMERNet/tree/main/cdm)

</div>

Welcome to the official repository of UniMERNet, a solution that converts images of mathematical expressions into LaTeX, suitable for a wide range of real-world scenarios.

## News 🚀🚀🚀
**2024.09.06** 🎉🎉  <font color="red">UniMERNet Update: The new version features a smaller model and faster inference. Training code is now open-sourced. For details, see the latest paper [UniMERNet](https://arxiv.org/abs/2404.15254).</font>   
**2024.09.06** 🎉🎉  <font color="red">Introducing a new metric for formula recognition: [CDM](https://github.com/opendatalab/UniMERNet/tree/main/cdm). Compared to BLEU/EditDistance, CDM provides a more intuitive and accurate evaluation score, allowing for fair comparison of different models without being affected by formula expression diversity. </font>  
**2024.07.21** 🎉🎉  Add Math Formula Detection (MFD) Tutorial based on [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit) MFD model.  
**2024.06.06** 🎉🎉  Open-sourced evaluation code for UniMER dataset.  
**2024.05.06** 🎉🎉  Open-sourced UniMER dataset, including UniMER-1M for model training and UniMER-Test for MER evaluation.  
**2024.05.06** 🎉🎉  Add Streamlit formula recognition demo and provided local deployment App.  
**2024.04.24** 🎉🎉  Paper now available on [ArXiv](https://arxiv.org/abs/2404.15254).  
**2024.04.24** 🎉🎉  Inference code and checkpoints have been released. 


## Demo Video
https://github.com/opendatalab/UniMERNet/assets/69186975/ac54c6b9-442c-48b0-95f9-a4a3fce8780b


https://github.com/opendatalab/UniMERNet/assets/69186975/09b71c55-c58a-4792-afc1-d5774880ccf8

## Quick Start

### Clone the repo and download the model
```bash
git clone https://github.com/opendatalab/UniMERNet.git
```

```bash
cd UniMERNet/models
# Download the model and tokenizer individually or use git-lfs
git lfs install
git clone https://huggingface.co/wanderkid/unimernet_base  # 1.3GB  
git clone https://huggingface.co/wanderkid/unimernet_small # 773MB  
git clone https://huggingface.co/wanderkid/unimernet_tiny  # 441MB  

# you can also download the model from ModelScope
git clone https://www.modelscope.cn/wanderkid/unimernet_base.git
git clone https://www.modelscope.cn/wanderkid/unimernet_small.git
git clone https://www.modelscope.cn/wanderkid/unimernet_tiny.git

```

### Installation

> Create a clean Conda environment

```bash
conda create -n unimernet python=3.10
conda activate unimernet
```

> Method 1: Install via pip (recommended for general users)

```bash
pip install -U "unimernet[full]"
```

> Method 2: Local installation (recommended for developers)

```bash
pip install -e ."[full]"
```





### Running UniMERNet

1. **Streamlit Application**: For an interactive and user-friendly experience, use our Streamlit-based GUI. This application allows real-time formula recognition and rendering.

    ```bash
    unimernet_gui
    ```
    Ensure you have the latest version of UniMERNet installed (`pip install --upgrade unimernet & pip install "unimernet[full]"`) for the streamlit GUI application.

2. **Command-line Demo**: Predict LaTeX code from an image.

    ```bash
    python demo.py
    ```

3. **Jupyter Notebook Demo**: Recognize and render formula from an image.

    ```bash
    jupyter-lab ./demo.ipynb
    ```


## Performance Comparison (BLEU) with SOTA Methods.

> UniMERNet significantly outperforms mainstream models in recognizing real-world mathematical expressions, demonstrating superior performance across Simple Printed Expressions (SPE), Complex Printed Expressions (CPE), Screen-Captured Expressions (SCE), and Handwritten Expressions (HWE), as evidenced by the comparative BLEU Score evaluation.  

![BLEU](./asset/papers/fig1_bleu.jpg)



## Visualization Result with Different Methods.

> UniMERNet excels in visual recognition of challenging samples, outperforming other methods.  

![Visualization](https://github.com/opendatalab/VIGC/assets/69186975/6edcac69-5082-43a2-8095-5681b7a707b9)

## UniMER Dataset
### Introduction
The UniMER dataset is a specialized collection curated to advance the field of Mathematical Expression Recognition (MER). It encompasses the comprehensive UniMER-1M training set, featuring over one million instances that represent a diverse and intricate range of mathematical expressions, coupled with the UniMER Test Set, meticulously designed to benchmark MER models against real-world scenarios. The dataset details are as follows:

**UniMER-1M Training Set:**
  - Total Samples: 1,061,791 Latex-Image pairs
  - Composition: A balanced mix of concise and complex, extended formula expressions
  - Aim: To train robust, high-accuracy MER models, enhancing recognition precision and generalization

**UniMER Test Set:**
  - Total Samples: 23,757, categorized into four types of expressions:
    - Simple Printed Expressions (SPE): 6,762 samples
    - Complex Printed Expressions (CPE): 5,921 samples
    - Screen Capture Expressions (SCE): 4,742 samples
    - Handwritten Expressions (HWE): 6,332 samples
  - Purpose: To provide a thorough evaluation of MER models across a spectrum of real-world conditions

### Dataset Download
You can download the dataset from [OpenDataLab](https://opendatalab.com/OpenDataLab/UniMER-Dataset) (recommended for users in China) or [HuggingFace](https://huggingface.co/datasets/wanderkid/UniMER_Dataset).

### Download UniMER-Test Dataset


Download the UniMER-1M dataset and extract it to the following directory:
```bash
./data/UniMER-1M
```

Download the UniMER-Test dataset and extract it to the following directory:
```bash
./data/UniMER-Test
```

## Training

To train the UniMERNet model, follow these steps:

1. **Specify the Training Dataset Path**: Open the `configs/train` fold and set the path to your training dataset.

2. **Run the Training Script**: Execute the following command to start the training process.

    ```bash
    bash script/train.sh
    ```

### Notes:
- Ensure that the dataset path specified in the `configs/train` fold is correct and accessible.
- Monitor the training process for any errors or issues.

## Testing

To test the UniMERNet model, follow these steps:

1. **Specify the Test Dataset Path**: Open the `configs/val` fold and set the path to your test dataset.

2. **Run the Test Script**: Execute the following command to start the testing process.

    ```bash
    bash script/test.sh
    ```

### Notes:
- Ensure that the dataset path specified in the `configs/val` fold is correct and accessible.
- The `test.py` script will use the specified test dataset for evaluation. Remember to change the test set path in test.py to your actual path.
- Review the test results for performance metrics and any potential issues.

## Math Formula Detection Tutorial

The prerequisite for formula recognition is to detect the areas within PDF or webpage screenshots where formulas are located. The [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit) includes a powerful model for detecting formulas. If you wish to perform both formula detection and recognition by yourself, you can refer to the [Formula Detection Tutorial](./MFD/README.md) for guidance on deploying and using the formula detection model.


## TODO

[✅] Release inference code and checkpoints of UniMERNet.  
[✅] Release UniMER-1M and UniMER-Test.  
[✅] Open-source the Streamlit formula recognition GUI application.   
[✅] Release the training code for UniMERNet.  

## Citation
If you find our models / code / papers useful in your research, please consider giving us a star ⭐ and citing our work 📝, thank you :)
```bibtex
@misc{wang2024unimernetuniversalnetworkrealworld,
      title={UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition}, 
      author={Bin Wang and Zhuangcheng Gu and Guang Liang and Chao Xu and Bo Zhang and Botian Shi and Conghui He},
      year={2024},
      eprint={2404.15254},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2404.15254}, 
}

@misc{wang2024cdmreliablemetricfair,
      title={CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation}, 
      author={Bin Wang and Fan Wu and Linke Ouyang and Zhuangcheng Gu and Rui Zhang and Renqiu Xia and Bo Zhang and Conghui He},
      year={2024},
      eprint={2409.03643},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2409.03643}, 
}
```

## Acknowledgements
- [VIGC](https://github.com/opendatalab/VIGC). The model framework is dependent on VIGC.
- [Texify](https://github.com/VikParuchuri/texify). A mainstream MER algorithm, UniMERNet data processing refers to Texify.
- [Latex-OCR](https://github.com/lukas-blecher/LaTeX-OCR). Another mainstream MER algorithm.
- [Donut](https://huggingface.co/naver-clova-ix/donut-base). The UniMERNet's Transformer Encoder-Decoder are referenced from Donut.
- [Nougat](https://github.com/facebookresearch/nougat). The tokenizer uses Nougat.

## Contact Us
If you have any questions, comments, or suggestions, please do not hesitate to contact us at wangbin@pjlab.org.cn.

## License
[Apache License 2.0](LICENSE)
