Metadata-Version: 2.1
Name: fast-keywords
Version: 0.1.0
Summary: Fast keyword identification with n-gram vector string matching.
Home-page: https://github.com/christianj6/fast-keyword-identification
Author: Christian Johnson
Author-email: 
License: unlicensed
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Description-Content-Type: text/markdown
Requires-Dist: pandas (==1.2.3)
Requires-Dist: scikit-learn (==0.23.0)
Requires-Dist: sparse-dot-topn (==0.2.9)
Requires-Dist: dill (==0.3.1.1)
Requires-Dist: nltk (==3.4.5)
Requires-Dist: tqdm (==4.47.0)

### Fast Keyword Identification
Fast keyword identification with n-gram vector string matching.
***

#### Overview
This package provides a generic pipeline for fuzzy-identification of keywords in large document collections. For example, if you wish to find all occurrences of the keyword "Walmart" in a large document collection, but expect some typos or variations in spelling, this module will allow you to quickly identify all matches. The matcher is based on a character n-gram vector model rather than the slower string edit distance. The module is originally intended for brand monitoring applications.

***

### Installation

```
pip install fast-keywords
```

***

### CLI

```
python -m fast_keywords --help
```

***

### Usage

```
python -m fast_keywords -k keywords.txt -c corpus.csv -l english -b 0.75
```

***

### Training Models for Additional Filtering

While the main script will search for keywords in the provided corpus, filtering according to the match confidence, you can also train and use simple text classifiers as an additional filter to remove dubious matches. For example, if you are searching for the company "apple," but find your searches frequently return references to fruit, you can train a model which will exclude those matches based on the surrounding text of matched keywords. Instructions for model training and usage are provided below.

1. After searching for keywords you will find a column "Match is Invalid" in the ```output.xlsx``` file.

2. Modify this column, changing matches which should be filtered out to "1".

3. Train a new model using the ```--train``` flag, providing the modified ```output.xlsx``` file and the original keywords file, as in the command below.

   1. ```
      python -m fast_keywords --train -d output.xlsx -k keywords.txt
      ```

4. The train command will create a directory with several ```model.pb``` files which you can distribute and use for filtering. You should use the absolute path to this containing directory as the model path passed with the ```-m``` flag.

5. You can use your models when predicting as in the below command. You can also pass previously-trained models using the ```-m``` flag to continue training on new data when running the train command.

```
python -m fast_keywords -k keywords.txt -c corpus.csv -l english -b 0.75 -m model.pb
```

***

### Notes

- Your input .csv must have a "text" column containing documents.
- The main script will create a a file ```output.xlsx``` summarizing identified keywords and their metadata.


