Metadata-Version: 2.1
Name: clean-text-my
Version: 0.1.1
Home-page: https://github.com/malaysia-ai/clean_text_my
Author: Malaysia AI
Author-email: malaysia.ai2020@gmail.com
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: aiohttp ==3.8.5
Requires-Dist: aiosignal ==1.3.1
Requires-Dist: async-timeout ==4.0.3
Requires-Dist: attrs ==23.1.0
Requires-Dist: certifi ==2023.7.22
Requires-Dist: charset-normalizer ==3.2.0
Requires-Dist: Cython ==3.0.0
Requires-Dist: datasets ==2.14.4
Requires-Dist: dill ==0.3.7
Requires-Dist: filelock ==3.12.2
Requires-Dist: frozenlist ==1.4.0
Requires-Dist: fsspec ==2023.6.0
Requires-Dist: huggingface-hub ==0.16.4
Requires-Dist: idna ==3.4
Requires-Dist: multidict ==6.0.4
Requires-Dist: multiprocess ==0.70.15
Requires-Dist: numpy ==1.25.2
Requires-Dist: packaging ==23.1
Requires-Dist: pandas ==2.0.3
Requires-Dist: pyarrow ==13.0.0
Requires-Dist: python-dateutil ==2.8.2
Requires-Dist: pytz ==2023.3
Requires-Dist: PyYAML ==6.0.1
Requires-Dist: requests ==2.31.0
Requires-Dist: six ==1.16.0
Requires-Dist: tqdm ==4.66.1
Requires-Dist: typing-extensions ==4.7.1
Requires-Dist: tzdata ==2023.3
Requires-Dist: urllib3 ==2.0.4
Requires-Dist: xxhash ==3.3.0
Requires-Dist: yarl ==1.9.2
Requires-Dist: scipy ==1.11.2
Requires-Dist: rich ==13.5.2

# clean-text-my

## Description
The `clean-text-my` repository contains a Python script that performs a deduplication process on a text dataset. This process is implemented based on the code provided in the [Jupyter Notebook](https://github.com/malaysia-ai/text-dataset-dedup).

## Installing from the PyPI
```console
$ pip install clean-text-my
```
