Metadata-Version: 2.1
Name: tfidf-zones
Version: 1.2.1
Summary: TF-IDF zone analysis CLI — classify terms into too-common, goldilocks, and too-rare zones
License: MIT
Keywords: tfidf,tf-idf,nlp,text-analysis,zones
Author: Craig Trim
Author-email: craigtrim@gmail.com
Maintainer: Craig Trim
Maintainer-email: craigtrim@gmail.com
Requires-Python: >=3.11,<3.14
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: rich (>=13.0,<14.0)
Requires-Dist: scikit-learn (>=1.5,<2.0)
Requires-Dist: wordnet-lookup (>=1.2)
Description-Content-Type: text/markdown

# tfidf-zones

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![Poetry](https://img.shields.io/badge/packaging-poetry-cyan.svg)](https://python-poetry.org/)
[![scikit-learn](https://img.shields.io/badge/scikit--learn-1.5-orange.svg)](https://scikit-learn.org/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)]()
[![Downloads](https://img.shields.io/pepy/dt/tfidf-zones)](https://pepy.tech/project/tfidf-zones)
[![Downloads/Month](https://img.shields.io/pepy/dm/tfidf-zones)](https://pepy.tech/project/tfidf-zones)

CLI tool that classifies terms in text documents into three zones based on TF-IDF and document frequency:

- **Too Common** — high document frequency (df > 0.2N)
- **Goldilocks** — high TF-IDF score within a moderate DF band (3 ≤ df ≤ 0.2N, tfidf ≥ Q95)
- **Too Rare** — low document frequency (df < 3)

Useful for stylometric analysis, authorship attribution, and understanding term importance.

## Install

```bash
poetry install
```

## Usage

```bash
# Analyze a single file
poetry run tfidf-zones --file novel.txt --output results.csv

# Use scikit-learn engine with bigrams
poetry run tfidf-zones --file novel.txt --scikit --ngram 2 --output results.csv

# Analyze a directory of .txt files
poetry run tfidf-zones --dir ./texts/ --output results.csv

# Show top 25 terms per zone with custom chunk size
poetry run tfidf-zones --file novel.txt --top-k 25 --chunk-size 500 --output results.csv
```

## Recipes

**Find content-word bigrams across a corpus:**

```bash
poetry run tfidf-zones \
  --dir ./texts/ --limit 100 --output results.csv \
  --no-chunk --wordnet --ngram 2 --no-ngram-stopwords
```

Combines `--wordnet` (only real English words), `--ngram 2` (bigrams), and `--no-ngram-stopwords` (discard bigrams containing stop/function words like "of_the") to surface meaningful two-word terms.

**Find content phrases (trigrams and above):**

```bash
poetry run tfidf-zones \
  --dir ./texts/ --output results.csv \
  --no-chunk --wordnet --ngram 3 --no-ngram-stopwords
```

Increase `--ngram` to 3, 4, or 5 to find longer phrases. The stopword filter removes any n-gram where at least one token is a stop word or function word, so only content-rich phrases survive.

**Corpus analysis with post-processing filters:**

```bash
poetry run tfidf-zones \
  --dir ./texts/ --output results.csv \
  --no-chunk --wordnet --min-df 2 --min-tf 2
```

Use `--min-df` and `--min-tf` to remove terms that appear in too few documents or have too few total occurrences, reducing noise from hapax legomena.

## Options

| Flag | Default | Description |
|------|---------|-------------|
| `--file` | | Path to a single text file |
| `--dir` | | Path to a directory of `.txt` files |
| `--scikit` | off | Use scikit-learn TF-IDF engine (default: pure Python) |
| `--top-k` | `10` | Number of terms per zone |
| `--ngram` | `1` | N-gram level (1–5, or 6 for skipgrams) |
| `--chunk-size` | `2000` | Tokens per chunk (min 100) |
| `--limit` | all | Randomly select N files from directory (requires `--dir`) |
| `--no-chunk` | off | Each file = one document, no chunking (requires `--dir`) |
| `--wordnet` | off | Only recognized English words participate in TF-IDF |
| `--no-ngram-stopwords` | off | Discard n-grams containing stop/function words (requires `--ngram` ≥ 2) |
| `--min-df` | | Remove terms with document frequency below this value |
| `--min-tf` | | Remove terms with term frequency below this value |
| `--output` | | Output CSV file path (required) |

Either `--file` or `--dir` is required (not both).

## How It Works

Text is tokenized, split into chunks, and scored with TF-IDF. Chunking a single document into sub-documents prevents IDF from collapsing to a constant. Terms are then bucketed into zones by their document-frequency percentile.

Two engines are available: a pure-Python implementation and a scikit-learn backed implementation. Both use smooth IDF (`log((1+N)/(1+DF)) + 1`) and produce comparable results.

