Metadata-Version: 2.1
Name: shoten
Version: 0.1.0
Summary: ...
Home-page: https://github.com/adbar/shoten
Author: Adrien Barbaresi
Author-email: barbaresi@bbaw.de
License: GPLv3+
Keywords: nlp
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.6
Requires-Dist: courlan (>=0.6.0)
Requires-Dist: htmldate (>=1.2.0)
Requires-Dist: numpy
Requires-Dist: scikit-learn (>=0.24.2)
Requires-Dist: simplemma (>=0.5.0)

======
Shoten
======


.. image:: https://img.shields.io/pypi/v/shoten.svg
    :target: https://pypi.python.org/pypi/shoten
    :alt: Python package

.. image:: https://img.shields.io/pypi/pyversions/shoten.svg
    :target: https://pypi.python.org/pypi/shoten
    :alt: Python versions

.. image:: https://img.shields.io/codecov/c/github/adbar/shoten.svg
    :target: https://codecov.io/gh/adbar/shoten
    :alt: Code Coverage


Helper functions to find word trends (i.e. extract tokens, lemmatize and filter).


Installation
------------

``pip/pip3 install -U git+https://github.com/adbar/shoten.git``


Usage
-----


Input
^^^^^

Two possibilities for input data:

- XML-TEI files as generated by `trafilatura <https://trafilatura.readthedocs.io/>`_:
    1. ``from shoten import gen_wordlist``
    2. ``myvocab = gen_wordlist(mydir, ['de', 'en'])``
- TSV-file contaning a word list: word form + ``TAB`` + date (``YYYY-MM-DD`` format) + possible 3rd column (source)
    1. ``from shoten import load_wordlist``
    2. ``myvocab = load_wordlist(myfile, ['de', 'en'])``

Language codes: optional list of languages to be considered for lemmatization, ordered by relevance. `ISO 639-1 codes <https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes>`_, see the `list of supported languages <https://github.com/adbar/simplemma>`_.

Optional argument ``maxdiff``: maximum number of days to consider (default: 1000, i.e. going back up to 1000 days from today).


Filters
^^^^^^^

``from shoten.filters import *``


- ``hapax_filter(myvocab, freqcount=2)``: (default frequency: <= 2)
- ``shortness_filter(myvocab, threshold=20)``: length threshold in percent of word lengths
- ``frequency_filter(myvocab, max_perc=50, min_perc=.001)``: maximum and minimum frequencies in percent
- ``oldest_filter(myvocab, threshold=50)``: discard the oldest words (threshold in percent)
- ``freshness_filter(myvocab, percentage=10)``: keep the X% freshest words
- ``ngram_filter(myvocab, threshold=90, verbose=False)``: retains X% words based on character n-gram frequencies; runs out of memory if the vocabulary is too large (8 GB RAM recommended)
- ``sources_freqfilter(myvocab, threshold=2)``: remove words which are only present in less than x sources
- ``sources_filter(myvocab, myset)``: only keep the words for which the source contains a string listed in the input set
- ``wordlist_filter(myvocab, mylist, keep_words=False)``: keep or discard words present in the input list


Reduce vocabulary size with a filter:

``myvocab = oldest_filter(myvocab)``

They can be chained:

``myvocab = oldest_filter(shortness_filter(myvocab))``


Output
^^^^^^

.. code-block:: python

    # print one-by-one
    for word in sorted(myvocab):
        print(word)
    # transfer to a list
    results = [w for w in myvocab]


CLI
^^^

``shoten --help``


Additional information
----------------------

*Shoten* = focal point in Japanese (`焦点 <https://en.wiktionary.org/wiki/%E7%84%A6%E7%82%B9#Japanese>`_).

Project webpage: `Webmonitor <https://www.dwds.de/d/korpora/webmonitor>`_.



