Metadata-Version: 2.1
Name: SEFR-CUT
Version: 0.1.dev0
Summary: UNKNOWN
Home-page: https://github.com/mrpeerat/SEFR_CUT
Author: mrpeerat
Author-email: peerat.l_s19@vistec.ac.th
License: MIT
Keywords: thai word segmentation deep learning machine learning neural network development
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Description-Content-Type: text/markdown
Requires-Dist: tensorflow (>=2.0.0)
Requires-Dist: pandas
Requires-Dist: scipy
Requires-Dist: numpy
Requires-Dist: scikit-learn
Requires-Dist: python-crfsuite
Requires-Dist: pyahocorasick

# SEFR CUT
Domain Adaptation of Thai Word Segmentation Models using Stacked Ensemble (EMNLP 2020) <br>
CRF as Stacked Model and DeepCut as Baseline model<br>

## Read more:
- Paper: [Domain Adaptation of Thai Word Segmentation Models using Stacked Ensemble]()
- Blog: [Domain Adaptation กับตัวตัดคำ มันดีย์จริงๆ](https://medium.com/@pingloaf)

## Install
> pip install sefr_cut

## How To use
### Requirements
- python >= 3.6
- python-crfsuite >= 0.9.7
- pyahocorasick == 1.4.0

## Example
You can play the example on [SEFR Example notebook](https://github.com/mrpeerat/SEFR_CUT/blob/master/Notebooks/1.SEFR_CUT%20example.ipynb)
### Load Engine & Engine Mode
- ws1000, tnhc
  - ws1000: Model trained on Wisesight-1000 and test on Wisesight-160
  - tnhc: Model trained on TNHC (80:20 train&test split with random seed 42)
  - BEST: Trained on BEST-2010 Corpus (NECTEC)
  ```
  SEFR_CUT.load_model(engine='ws1000')
  # OR
  SEFR_CUT.load_model(engine='tnhc')
  # OR
  SEFR_CUT.load_model(engine='best')
  ```
- tl-deepcut-XXXX
  - We also provide transfer learning of deepcut on 'Wisesight' as tl-deepcut-ws1000 and 'TNHC' as tl-deepcut-tnhc
  ```
  SEFR_CUT.load_model(engine='tl-deepcut-ws1000')
  # OR
  SEFR_CUT.load_model(engine='tl-deepcut-tnhc')
  ```
- deepcut
  - We also provide the original deepcut
  ```
  SEFR_CUT.load_model(engine='deepcut')
  ```
### Segment Example
- Segment with default k
  ```
  SEFR_CUT.load_model(engine='ws1000')
  print(sefr_cut.tokenize(['สวัสดีประเทศไทย','ลุงตู่สู้ๆ']))
  print(sefr_cut.tokenize(['สวัสดีประเทศไทย']))
  print(sefr_cut.tokenize('สวัสดีประเทศไทย'))

  [['สวัสดี', 'ประเทศ', 'ไทย'], ['ลุง', 'ตู่', 'สู้', 'ๆ']]
  [['สวัสดี', 'ประเทศ', 'ไทย']]
  [['สวัสดี', 'ประเทศ', 'ไทย']]
  ```
- Segment with different k
  ```
  SEFR_CUT.load_model(engine='ws1000')
  print(sefr_cut.tokenize(['สวัสดีประเทศไทย','ลุงตู่สู้ๆ'],k=5)) # refine only 5% of character number
  print(sefr_cut.tokenize(['สวัสดีประเทศไทย','ลุงตู่สู้ๆ'],k=100)) # refine 100% of character number

  [['สวัสดี', 'ประเทศไทย'], ['ลุงตู่', 'สู้', 'ๆ']]
  [['สวัสดี', 'ประเทศ', 'ไทย'], ['ลุง', 'ตู่', 'สู้', 'ๆ']]
  ```

## Evaluation
- Character & Word Evaluation is provided by call fuction ```evaluation()```
  - For example
  ```

  ```

## Performance
<img src="https://user-images.githubusercontent.com/21156980/94403460-aa131980-0197-11eb-9a68-8e2927a5059d.PNG" width="569" height="420" />
<img src="https://user-images.githubusercontent.com/21156980/94404121-ac29a800-0198-11eb-9ce5-898d2e4e6a82.PNG" width="569" height="420" />
<img src="https://user-images.githubusercontent.com/21156980/94404126-ae8c0200-0198-11eb-9151-6adad36cfa31.PNG" width="569" height="341" />

## How to re-train?
- You can re-train model in folder [Notebooks](https://github.com/mrpeerat/SEFR_CUT/tree/master/Notebooks) We provided everything for you!!
  ### Re-train Model
  - You need to XXXXXXXXXXX
  - Link:[HERE](https://github.com/mrpeerat/SEFR_CUT/blob/master/Notebooks/2.Train_DS_model.ipynb)
  ### Filter and Refine Example
  - You need to XXXXXXXXXXX
  - Link:[HERE](https://github.com/mrpeerat/SEFR_CUT/blob/master/Notebooks/3.Stacked%20Model%20Example.ipynb)
  ### Use your own model?
  - You need to XXXXXXXXXXX

## Citation
- Wait our paper shown in ACL Anthology

Thank you many code from

- [Deepcut](https://github.com/rkcosmos/deepcut) (Baseline Model) : We used some of code from Deepcut to perform transfer learning 
- [@bact](https://github.com/bact) (CRF training code) : We used some from https://github.com/bact/nlp-thai in training CRF Model




