Metadata-Version: 2.4
Name: filenthropist
Version: 0.3.1
Summary: Local-first file scanner and labeler that lets you work safely with autonomous AI agents. Multilingual, currently focused on Dutch PII and GDPR.
Project-URL: Homepage, https://github.com/LokaalHub/filenthropist
Project-URL: Repository, https://github.com/LokaalHub/filenthropist
Project-URL: Issues, https://github.com/LokaalHub/filenthropist/issues
Project-URL: Changelog, https://github.com/LokaalHub/filenthropist/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/LokaalHub/filenthropist#readme
Author: LokaalHub
License-Expression: MIT
License-File: LICENSE
Keywords: ai-agents,ai-safety,autonomous-ai,compliance,data-governance,dutch,file-scanner,gdpr,local-first,multilingual,ner,pii,privacy
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Legal Industry
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: General
Classifier: Topic :: Utilities
Requires-Python: >=3.11
Requires-Dist: chardet>=7.0
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: lxml>=5.0
Requires-Dist: python-calamine>=0.3
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: watchdog>=4.0
Provides-Extra: all
Requires-Dist: fastapi>=0.111; extra == 'all'
Requires-Dist: huggingface-hub>=0.20; extra == 'all'
Requires-Dist: pillow>=10.0; extra == 'all'
Requires-Dist: pymupdf>=1.24; extra == 'all'
Requires-Dist: pytesseract>=0.3; extra == 'all'
Requires-Dist: python-multipart>=0.0.9; extra == 'all'
Requires-Dist: rapidocr-onnxruntime>=1.4; extra == 'all'
Requires-Dist: torch>=2.2; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Requires-Dist: uvicorn>=0.29; extra == 'all'
Provides-Extra: dev
Requires-Dist: faker>=24.0; extra == 'dev'
Requires-Dist: jiwer>=3.0; extra == 'dev'
Requires-Dist: memory-profiler>=0.61; extra == 'dev'
Requires-Dist: openpyxl>=3.1; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-benchmark>=4.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: python-docx>=1.1; extra == 'dev'
Requires-Dist: reportlab>=4.1; extra == 'dev'
Provides-Extra: gliner
Requires-Dist: gliner>=0.2; extra == 'gliner'
Provides-Extra: ner
Requires-Dist: huggingface-hub>=0.20; extra == 'ner'
Requires-Dist: torch>=2.2; extra == 'ner'
Requires-Dist: transformers>=4.40; extra == 'ner'
Provides-Extra: ocr
Requires-Dist: pillow>=10.0; extra == 'ocr'
Requires-Dist: pymupdf>=1.24; extra == 'ocr'
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Requires-Dist: rapidocr-onnxruntime>=1.4; extra == 'ocr'
Provides-Extra: web
Requires-Dist: fastapi>=0.111; extra == 'web'
Requires-Dist: python-multipart>=0.0.9; extra == 'web'
Requires-Dist: uvicorn>=0.29; extra == 'web'
Description-Content-Type: text/markdown

# Filenthropist

**The local-first file scanner and labeler that lets you work safely with autonomous AI agents. Multilingual by design, currently focused on Dutch PII and GDPR — so your governance tool never becomes a data breach.**

Filenthropist scans documents, detects PII, and labels each file with an access level that autonomous AI agents check before touching it. It runs entirely on your machine, never sends data to the cloud, and exposes a single HTTP API endpoint for agents to consult. The architecture is language-agnostic; the current release ships with Dutch-focused detection and GDPR-aware reporting, with more languages planned.

---

## Why Filenthropist?

- **Local-only processing** — files never leave your machine. No cloud APIs, no data uploads, no telemetry.
- **Dutch-focused PII detection today** — built for BSN, IBAN, Dutch addresses, KvK numbers, and NL-specific identifiers. Not a US-centric tool with a language pack bolted on. The architecture is language-agnostic and a multilingual model is already available in the registry — see [`docs/MODELS.md`](docs/MODELS.md).
- **Redacted previews only** — the database stores redacted text, not raw PII. Even if someone accesses the SQLite file, they get `[NAAM]` and `[BSN]`, never the personal data.
- **AI agent gateway** — one API call (`/api/can-access`) tells any agent whether a file is safe to use.
- **GDPR Article 30 RoPA export** — generates a verwerkingsregister directly from your scanned files, with Dutch legal bases pre-filled per document type.
- **Advisory-only** — Filenthropist labels and advises. It never deletes, modifies, or moves your files.

---

## System requirements

- **Python 3.11 or newer**
- **Tesseract 5** with the Dutch language pack (needed for OCR on scanned PDFs and images)
- **~1 GB free disk** for a PII NER model (downloaded on first run)

### Install Tesseract

macOS (Homebrew):

```bash
brew install tesseract tesseract-lang
```

Debian / Ubuntu:

```bash
sudo apt install tesseract-ocr tesseract-ocr-nld
```

For a step-by-step guide covering Python, Tesseract, Windows, and common errors, see [`docs/INSTALL.md`](docs/INSTALL.md).

---

## Install

We recommend [pipx](https://pipx.pypa.io) so Filenthropist lives in its own isolated environment:

```bash
pipx install "filenthropist[all]"
```

The base install is intentionally minimal. Pick the extras that match what you need:

```bash
pipx install filenthropist              # CLI only, no OCR/NER/web
pipx install "filenthropist[ocr]"       # + Tesseract OCR
pipx install "filenthropist[ner]"       # + Dutch/multilingual PII NER
pipx install "filenthropist[web]"       # + local dashboard
pipx install "filenthropist[all]"       # everything (recommended)
```

---

## Quick start

Four commands take you from a fresh install to a usable dashboard:

```bash
filenthropist doctor                    # verify environment
filenthropist init                      # interactive model picker
filenthropist scan ~/Documents          # run first scan
filenthropist serve                     # open web dashboard
```

- **`doctor`** — checks Python version, Tesseract install (with Dutch pack), writable config directory, and reports anything missing so you can fix it before scanning.
- **`init`** — walks you through selecting a PII NER model based on your language and priority (speed, balanced, accuracy). Downloads the model and writes your config.
- **`scan <path>`** — walks the directory, extracts text (including OCR for scanned PDFs), classifies each document, detects PII, and stores redacted labels in `~/.filenthropist/filenthropist.db`.
- **`serve`** — starts the local dashboard at `http://localhost:8080` for reviewing labels and making retention decisions.

For three persona-based walkthroughs (freelancer, SME, multilingual org), see [`docs/QUICKSTART.md`](docs/QUICKSTART.md).

---

## Choosing a PII model

Filenthropist supports multiple PII NER models so you can trade off language coverage, accuracy, and speed.

- **Recommended for Dutch documents**: [`LokaalHub/nl-lokaal-middel`](https://huggingface.co/LokaalHub/nl-lokaal-middel) — F1 0.84 on the ai4privacy Dutch validation set. Detects Dutch names, addresses, BSN-in-context, IBAN, phone, email, and more. A faster/smaller sibling [`LokaalHub/nl-lokaal-klein`](https://huggingface.co/LokaalHub/nl-lokaal-klein) (F1 0.78, ~180 MB) is the default for laptops and the `combined` provider.
- **Multilingual and English-only options** are available for mixed-language or English-only corpora.
- **Speed vs. accuracy** — each model is tagged with a priority tier (`fast`, `balanced`, `accuracy`) so you can match it to your hardware.

Browse and pick interactively:

```bash
filenthropist init                      # wizard: language + priority
filenthropist models list               # show every model in the registry
filenthropist models info <model-id>    # details for one model
```

Full decision tree and registry docs: [`docs/MODELS.md`](docs/MODELS.md).

---

## Scanning & querying

```bash
# Show all files with sensitive PII
filenthropist query --access-level sensitive_restricted

# Export all labels as JSON
filenthropist export --format json --output labels.json

# Export a GDPR Article 30 verwerkingsregister
filenthropist ropa --format csv --output verwerkingsregister.csv
```

The web dashboard (`filenthropist serve`) exposes the same data plus a review workflow for non-technical users.

AI agents integrate via a local HTTP API — `GET /api/can-access?path=...` returns an allow/deny decision, and `GET /api/redacted?path=...` returns the document text with PII replaced by type labels (`[NAAM]`, `[BSN]`, `[IBAN]`). See [`docs/AGENT_INTEGRATION.md`](docs/AGENT_INTEGRATION.md) for the full endpoint list and integration patterns.

---

## Configuration

On first run, Filenthropist writes `~/.filenthropist/config.yaml`. Edit it to tune scan behaviour, PII provider, and retention:

```yaml
scan:
  ignore_patterns: [".git", "node_modules", "__pycache__", ".venv"]
  max_file_size_mb: 100

pii:
  provider: "combined"         # "regex", "ner", "combined", "http", or "stub"
  ner_model_id: "LokaalHub/nl-lokaal-middel"

labeling:
  retention_policy_years: 2

classification:
  zeroshot_enabled: true
```

The `combined` provider runs regex detectors (BSN, IBAN, phone, email, postcode) alongside the NER model — structured PII that NER often misses is still caught.

---

## Privacy & security

- **Fully local.** No network calls except model downloads, which you can pre-cache for offline use.
- **Advisory-only.** Filenthropist never deletes, moves, or modifies your files.
- **Redacted previews.** The database stores redacted text, so compromising the DB does not leak raw PII.

For the threat model and hardening recommendations, see [`SECURITY.md`](SECURITY.md).

---

## License

MIT
