Metadata-Version: 2.1
Name: corpus-similarity
Version: 1.0.1
Summary: Measuring corpus similarity in Python
Home-page: https://github.com/jonathandunn/corpus_similarity
Author: Jonathan Dunn, Haipeng Li
Author-email: jonathan.dunn@canterbury.ac.nz
License: GNU GENERAL PUBLIC LICENSE v3
Keywords: text analytics,natural language processing,computational linguistics,corpus,corpora,similarity
Platform: UNKNOWN
Description-Content-Type: text/markdown
Requires-Dist: cytoolz
Requires-Dist: numpy
Requires-Dist: clean-text
Requires-Dist: scipy
Requires-Dist: sklearn

# corpus_similarity

Measure the similarity between two corpora (text datasets). The measures work best when each corpus is at least 10k words.

    from corpus_similarity import Similarity
    cs = Similarity(language = "eng")

    result = cs.calculate(corpus1, corpus2)

The package contains all preprocessing and training. Only the language needs to be specified. A list of supported languages is provided below.

# Input

The **Similarity.calculate** method requires two input corpora. These can be a list of strings or a filename (supports .txt and .gz files).

# Output

The output is a scalar measure of how similar the two corpora are. The values fall between 0 (very different) and 1 (very similar). The values are consistent within languages, but not across languages. For example, Swedish has higher relative similarity than Estonian.

# Installation

    pip install corpus_similarity

    pip install git+https://github.com/jonathandunn/corpus_similarity.git

# Languages

**Pacific Languages**

haw,    Hawaiian (Polynesian)

mri,    te reo (Polynesian)

smo,    Samoan (Polynesian)

ton,    Tongan (Polynesian)

ceb,    Cebuano (Austronesian)

mlg,    Malagasy (Austronesian)

msa,    Malay (Austronesian)

tgl,    Tagalog (Austronesian)

**Other Languages**

vie,	Vietnamese

ind,	Indonesian

tgl,	Tagalog

tam,	Tamil

tel,	Telugu

bul,	Bulgarian

ces,	Czech

lav,	Latvian

pol,	Polish

rus,	Russian

slv,	Slovenian

ukr,	Ukrainian

dan,	Danish

deu,	German

eng,	English

nld,	Dutch

nor,	Norwegian

swe,	Swedish

ell,	Greek

fas,	Farsi

hin,	Hindi

urd,	Urdu

cat,	Catalan

fra,	French

glg,	Galician

ita,	Italian

por,	Portuguese

ron,	Romanian

spa,	Spanish

jpn,	Japanese

kor,	Korean

ara,	Arabic

heb,	Hebrew

zho,	Chinese

tha,	Thai

tur,	Turkish

est,	Estonian

fin,	Finnish

hun,	Hungarian


