Metadata-Version: 2.1
Name: nepali-embedding
Version: 0.1
Summary: UNKNOWN
Home-page: https://github.com/pudasainishushant/nepali_embedding
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.5
Description-Content-Type: text/markdown
License-File: LICENSE.md
License-File: AUTHORS.md
Requires-Dist: transformers (==4.15.0)
Requires-Dist: scipy
Requires-Dist: torch
Requires-Dist: gensim (==4.1.2)
Requires-Dist: icu (==0.0.1)
Requires-Dist: scikit-learn (==0.24.2)

# nepali_embedding

Generate text embedding for nepali text

# Objective

Any NLP downstream tasks such as text classification, Named Entity Recognition performed on nepali textual data requires embedding of the words as the feature embedding matrix.
For english text, there are several ways we can generate embedding such as Word2Vec, Glove, BERT and so on from several open source NLP libraries. However, there seems to be no open source library for such tasks in nepali text. We aim to provide accurate word embedding for nepali text through several NLP based architectures so that it can be used in further NLP downstream tasks in nepali language.

# Models used overview

We have developed state of the art models for embedding generation for nepali text.

- Sabda2Vec
- Bakya2Vec
- NepBERT

# Usage

## Word_embedding

Use this script to get embedding of a word (sabda).

```python
from nepali_embedding.sabda2vec.inference import Sabda2Vec
sabda2vec_obj = Sabda2Vec(model_name = "sabda2vec_sm")
#Get embedding of the token
embedding = sabda2vec_obj.get_embedding("हार")
# Get top similar tokens
top_similar = sabda2vec_obj.get_most_similar("हार",5)
# Get similarity between two tokens
similarity_score = sabda2vec_obj.get_similarity_between_tokens("हार","पराजय")
```

```
    sabda_to_vec_model: https://www.dropbox.com/s/xkd29spkozoavhk/sabda_to_vec_model?dl=0
    sabda_to_vec_model_md: https://www.dropbox.com/s/55m5q4h5ys1l4np/sabda_to_vec_model_md?dl=0
```
## Loading embedding from nepBERT model
```python
from nepali_embedding.nepBERT.embedding_generator import NepBERT
nepbert = NepBERT()
test_sentence = 'आकाश भाई ज्ञानी मन्छे हो'
embedding = nepbert.get_bert_embedding_sentence(test_sentence,)
```


