Metadata-Version: 2.4
Name: metacls
Version: 0.4.0
Summary: Bulk metadata scrubbing for PDF/Office/image files — the remediation companion to MetaScout.
Author: Görkem Güler
License: MIT
Project-URL: Homepage, https://github.com/gorkemguler/MetaCLS
Project-URL: Repository, https://github.com/gorkemguler/MetaCLS
Project-URL: Changelog, https://github.com/gorkemguler/MetaCLS/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/gorkemguler/MetaCLS/issues
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Security
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: pikepdf>=9.0
Requires-Dist: olefile>=0.47
Requires-Dist: rich>=13.7
Requires-Dist: jinja2>=3.1
Requires-Dist: python-dotenv>=1.0
Requires-Dist: flask>=3.0
Requires-Dist: tomli>=2.0; python_version < "3.11"
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == "api"
Requires-Dist: uvicorn>=0.29; extra == "api"
Requires-Dist: python-multipart>=0.0.9; extra == "api"
Provides-Extra: image-fallback
Requires-Dist: Pillow>=10.0; extra == "image-fallback"
Requires-Dist: pillow-heif>=0.16; extra == "image-fallback"
Provides-Extra: media
Requires-Dist: mutagen>=1.47; extra == "media"
Provides-Extra: archive
Requires-Dist: py7zr>=0.20; extra == "archive"
Provides-Extra: msg
Requires-Dist: extract-msg>=0.48; extra == "msg"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: mutagen>=1.47; extra == "dev"
Requires-Dist: py7zr>=0.20; extra == "dev"
Requires-Dist: extract-msg>=0.48; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: fastapi>=0.110; extra == "dev"
Requires-Dist: uvicorn>=0.29; extra == "dev"
Requires-Dist: python-multipart>=0.0.9; extra == "dev"
Requires-Dist: Pillow>=10.0; extra == "dev"
Requires-Dist: pyftpdlib>=1.5; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/gorkemguler/MetaCLS/main/assets/banner.svg" alt="MetaCLS" width="100%">
</p>

<p align="center">
  <img alt="License" src="https://img.shields.io/badge/license-MIT-2dd4a7.svg">
  <img alt="Python" src="https://img.shields.io/badge/python-3.10%2B-2dd4a7.svg">
  <img alt="Platforms" src="https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-7dd88f.svg">
  <img alt="Status" src="https://img.shields.io/badge/status-active%20development-f6c454.svg">
</p>

<p align="center">
  Bulk metadata scrubbing for PDF, Office and image files.<br>
  Strip the metadata, keep the document, with a before/after proof report.
</p>

<p align="center">
  <sub>Pairs with <a href="https://github.com/gorkemguler/MetaScout">MetaScout</a>: MetaScout <i>finds</i> the leaks, MetaCLS <i>fixes</i> them.</sub>
</p>

<p align="center"><sub>🇬🇧 English · <a href="https://github.com/gorkemguler/MetaCLS/blob/main/README.tr.md">🇹🇷 Türkçe</a></sub></p>

<p align="center">
  <img src="https://raw.githubusercontent.com/gorkemguler/MetaCLS/main/assets/screenshot-report.png" alt="MetaCLS before/after report, 3 files, 18 metadata fields removed, 0 residual" width="90%">
</p>

---

## What is this?

An organisation runs MetaScout against its own site and finds a pile of published PDFs and
Office documents leaking author names, internal file paths, software/OS fingerprints and GPS
coordinates. Now someone has to actually **clean those files**. That's MetaCLS.

Point it at a folder (or drag files into the web UI, or POST them to the API) and it:

1. **scans** every supported file for embedded metadata,
2. **strips** it, aggressively by default, writing cleaned copies (originals untouched) or
   overwriting in place,
3. **verifies** each cleaned file by re-scanning it, and
4. **reports** exactly what was removed, per file, as JSON and a styled HTML page you can hand
   to a security team as evidence.

It only touches **metadata**: the document's visible content (body text, images, a scanned
signature) is never modified.

## What it removes

| Format | Engine | Removed |
| --- | --- | --- |
| **PDF** | [pikepdf](https://github.com/pikepdf/pikepdf) (QPDF) | `/Info` dictionary (Author, Title, Producer, Creator, CreationDate, …), the XMP metadata packet, `/PieceInfo` and other application-private data, page-level metadata, annotation authors + timestamps (`/T` `/M` `/CreationDate`), and the description + timestamps on embedded-file attachments. The file is **fully rewritten**, so values sitting in superseded cross-reference sections can't be recovered from the output. Encrypted PDFs need `--password`. With `--strip-form-values`: AcroForm field values and the XFA `<xfa:data>` packet too. |
| **Office** `.docx .xlsx .pptx` (+ macro-enabled `.docm .xlsm .pptm` and templates `.dotx .dotm .xltx .xltm .potx .potm`) | stdlib `zipfile` | `docProps/core.xml` (creator, lastModifiedBy, revision, timestamps), `docProps/app.xml` (Company, Manager, Template path), `docProps/custom.xml`, the embedded thumbnail, and Word revision-save-id fingerprints (`w:rsids`) from `settings.xml`. Dangling relationships and content-type overrides are pruned; per-member zip timestamps are normalised. `vbaProject.bin` is **kept** (breaking macros is worse than the small chance a name hides in it) and the report flags that the file was only partly scrubbed. |
| **OpenDocument** `.odt .ods .odp` | stdlib `zipfile` | `meta.xml`: initial-creator, creator, generator, editing-cycles/duration, timestamps, document statistics, user-defined fields, plus the `Thumbnails/` preview image (a rendered snapshot of the first page) and its `META-INF/manifest.xml` entry. |
| **Legacy Office** `.doc .xls .ppt` | `olefile` (pure Python) | The `\x05SummaryInformation` / `\x05DocumentSummaryInformation` property streams (author, last-saved-by, company, manager, template, title, timestamps, custom properties) are patched out **in place**: same file size, same format, same structure. `--in-place` works. If a container can't be parsed, MetaCLS falls back to a LibreOffice (`soffice`) re-render to `.docx/.xlsx/.pptx`. |
| **SVG** `.svg` | stdlib `xml` | `<metadata>` (RDF/Dublin-Core author/title/licence), `sodipodi:` / `inkscape:` / Adobe-Illustrator elements and attributes, and editor comments (`<!-- Created with … -->`). The drawing itself is untouched. |
| **Images** `.jpg .jpeg .png .gif .tif .tiff .heic .heif .webp` | [ExifTool](https://exiftool.org) | All EXIF / IPTC / XMP / GPS / MakerNotes, PNG/WebP text chunks and the JPEG/GIF comment block. The ICC colour profile and EXIF orientation are kept by default so the picture still renders correctly (`--no-keep-color-profile` / `--no-keep-orientation` to drop those too). HEIC also works via `pillow-heif` when exiftool is absent. |
| **Audio / video** `.mp3 .m4a .flac .ogg .opus .wav .aiff` / `.mp4 .mov .m4v .3gp .mkv .webm .avi` | `mutagen` / ExifTool / pure Python | Audio: all tags (ID3 / Vorbis / iTunes) and embedded cover art, `pip install 'metacls[media]'`. MP4-family video: exiftool clears the metadata atoms (`ItemList`, `Keys`, `UserData`, XMP, artist, `Make`/`Model` from a phone, GPS, `CreationDate`). **Matroska / WebM / AVI** (exiftool can't write these): the whole EBML `Tags` block and the `Info` title / dates / muxer-and-writer app names are blanked in place, overwritten with `Void` / `JUNK` padding of identical length, so the file length is unchanged and `--in-place` works; the track data is never touched. Not scanned by default, pass **`--media`** (or list the extensions in `--filetypes`). Spot-check playback. |
| **Containers** `.zip .eml .tar .tar.gz .tgz .tar.bz2 .tar.xz .7z .msg` | stdlib `zipfile` / `tarfile` / `email`; `py7zr` / `extract-msg` (optional) | With **`--recurse`**: each supported member of an archive, and each email attachment, is scrubbed with its own engine and the archive/message repacked. `.tar*` also has its per-member uid/gid/username/mtime headers normalised (a leak of the packer's identity). `.7z` needs `pip install 'metacls[archive]'`. `.msg` (Outlook) is **read-only**: `metacls inspect --recurse` lists what's inside (needs `metacls[msg]`); export to `.eml` to scrub. Non-scrubbable members pass through untouched. Nesting is followed (depth-limited). |

`--keep Title` (repeatable) spares a named field from the otherwise-aggressive strip.
`--backup` keeps `<name>.orig` next to an `--in-place` scrub.

Two **opt-in** flags go past metadata into identity data that's technically content:
`--strip-form-values` blanks PDF form values, AcroForm fields (`/V` `/DV`) and their cached
appearance, plus the `<xfa:data>` packet of an XFA form (the XFA template and schema are kept);
`--strip-office-authors` blanks Office tracked-change / comment **author names and dates**
(the change and comment text stays, so accept/reject still works).

## Install

```bash
pip install metacls                     # core: PDF + Office scrubbing
pip install 'metacls[api]'              # + the REST API service
pip install 'metacls[media]'            # + audio tag scrubbing (mutagen)
pip install 'metacls[archive]'          # + .7z recursion (py7zr)
pip install 'metacls[msg]'              # + read-only .msg inspection (extract-msg)
pip install 'metacls[image-fallback]'   # + Pillow (weak image fallback if exiftool is absent)
```

Image scrubbing needs the **exiftool** binary on `PATH`:

```bash
brew install exiftool                        # macOS
sudo apt install libimage-exiftool-perl      # Debian / Ubuntu
```

PDF, Office (modern **and** legacy `.doc/.xls/.ppt`), ODF and SVG scrubbing are pure Python
and need nothing extra. **LibreOffice** (`soffice`) is only used as a fallback for a legacy
container the in-place patcher can't parse.

> Python 3.10+ is supported. On a brand-new Python where `pikepdf` has no wheel yet, install
> under 3.12 instead.

## CLI

<p align="center">
  <img src="https://raw.githubusercontent.com/gorkemguler/MetaCLS/main/assets/screenshot-cli.svg" alt="metacls inspect and metacls clean in a terminal" width="90%">
</p>

### Inspect: see what's in the files (read-only)

```bash
metacls inspect ./published-docs
metacls inspect leak.pdf report.docx --json
```

Run this on the files MetaScout flagged to see exactly what they carry before you scrub.

### Clean

```bash
# default: originals untouched, cleaned copies written under ./metacls_cleaned/,
# mirroring the input tree, plus report.json + report.html
metacls clean ./published-docs

# overwrite the originals instead (asks first; -y to skip the prompt)
metacls clean ./published-docs --in-place

# preview only, change nothing
metacls clean ./published-docs --dry-run

# keep document titles, Turkish report
metacls clean ./published-docs --keep Title --report-lang tr

# just some files
metacls clean a.pdf b.docx c.jpg --out ./clean
```

Useful flags: `--filetypes`, `--no-recursive`, `--out DIR`, `--keep FIELD`, `--dry-run`,
`--no-verify`, `--no-keep-color-profile`, `--no-keep-orientation`, `--report-lang en|tr`,
`--password` (encrypted PDFs), `--strip-pdf-id`, `--backup` (keep `<name>.orig` with `--in-place`),
`--strip-form-values`, `--strip-office-authors` (opt-in, see above),
`--jobs N` (scrub N files in parallel), `--quarantine DIR` (overwrite the original but move
it to `DIR/<date>/` first, recoverable, safer than `--in-place`),
`--policy publish|internal|minimal` (named presets),
`--exclude GLOB` (repeatable, skip files/dirs when walking),
`--no-follow-symlinks` (don't scrub a symlinked file), `--progress` (a progress bar), and the
group-level `metacls --debug …` (re-raise on the first failing file instead of recording it).

`metacls inspect` takes `--media` and `--recurse` too, so you can point it at an archive or a
video and see what's inside before scrubbing.

**Project config:** a `.metacls.toml` in the working directory or a parent (up to the git
root) sets defaults per command; CLI flags and env vars still win.

```toml
[clean]
strip-office-authors = true
jobs = 4
keep = ["Title"]
```

**Exit codes** (so it works as a CI gate): `0` clean · `1` a file errored · `2` a cleaned file
still carried metadata on the verify re-scan · `3` (`--check` only) metadata found.

### Diff: track a directory over time

```bash
metacls clean ./published --out ./scan-jan     # once a month, into dated dirs
metacls clean ./published --out ./scan-feb
metacls diff ./scan-jan ./scan-feb             # what changed?
```

Shows files added/removed between the two runs and, the useful part, files where metadata
**reappeared** (someone re-saved the document in an editor). Exit code `1` if anything
regained metadata: drop it in a cron job.

### Watch: keep a drop folder scrubbed

```bash
metacls watch /srv/ftp/incoming --move-processed /srv/ftp/scrubbed --interval 10 --settle 5
metacls watch ./inbox --once                       # one pass, for cron
metacls watch ./inbox --pattern 'invoice-*.pdf' -j 4   # filter + parallel backlog
```

A poll loop for an FTP/SFTP landing zone: a file is only touched once it has stopped
changing for `--settle` seconds (a half-finished upload is never scrubbed), state lives in
`<dir>/.metacls-watch.json` so a restart doesn't reprocess everything, and a file
re-dropped with a newer timestamp is handled again. A `.metacls-watch.lock` file keeps a
second watcher off the same directory (a lock left by a dead process is stolen).
`--pattern GLOB` (repeatable) narrows what's picked up; `-j/--jobs N` scrubs a backlog in
parallel; state entries for files that have since vanished are pruned each pass. Scrubs in
place by default; `--to DIR` writes cleaned copies instead. A systemd template unit is in
[`platform/linux/`](https://github.com/gorkemguler/MetaCLS/blob/main/platform/linux/metacls-watch@.service).

## Web UI

```bash
metacls web           # opens http://127.0.0.1:8770/
```

Drag files onto the page, get them back scrubbed, individually or as a zip, with a
per-file before/after view and the full report. Past runs are listed under **History**.
Local, single-user, **no authentication**. Don't expose it to a network.

<p align="center">
  <img src="https://raw.githubusercontent.com/gorkemguler/MetaCLS/main/assets/screenshot-web.png" alt="MetaCLS local web UI, drag-and-drop file drop zone" width="90%">
</p>

The page also has a second form: **Or pull from FTP**. Point it at a host, remote
directory, and optional credentials, and MetaCLS fetches every matching file over
FTP or FTPS, scrubs it, and shows the same before/after report — no manual
download/upload round trip. The password is used once for that request and is
never stored or logged. An opt-in checkbox uploads the scrubbed copies straight
back to the same remote paths, overwriting the originals.

## REST API

For a Linux server, an FTP drop-box, or a CI pipeline that needs to hand files off to be
cleaned:

```bash
pip install 'metacls[api]'
metacls api           # http://127.0.0.1:8000/  ·  interactive docs at /docs
```

```bash
# what can this server scrub, and which optional tools are installed?
curl -sS http://127.0.0.1:8000/v1/formats

# submit  (recurse: look inside .zip/.tar/.7z/.eml · media: also audio/video)
curl -sS -X POST http://127.0.0.1:8000/v1/clean \
  -F files=@leak.pdf -F files=@report.docx -F report_lang=en -F recurse=true
# -> {"job_id": "...", "links": {...}}

# poll
curl -sS http://127.0.0.1:8000/v1/clean/<job_id>

# pull the cleaned files + report as a zip
curl -sSL -o cleaned.zip http://127.0.0.1:8000/v1/clean/<job_id>/download
```

Jobs run in a bounded background thread pool; `--max-workers` / `--max-pending` size it.
Uploads stream to disk with `--max-upload-mb` / `--max-files` caps. `GET /v1/formats` and
`GET /v1/health` are open even when an API key is set.

**Auth:** `metacls api --api-key KEY` (or `METACLS_API_KEY`) requires that key on every
`/v1` route except `/v1/health` and `/v1/formats`: send it as `X-API-Key: KEY` or
`Authorization: Bearer KEY`. Both `api` and `web` **refuse to bind a non-loopback host**
(`0.0.0.0`, a LAN IP) with no auth unless you pass `--insecure`; the supported way to expose
either is to bind `127.0.0.1` and put a reverse proxy in front.
**[docs/reverse-proxy.md](https://github.com/gorkemguler/MetaCLS/blob/main/docs/reverse-proxy.md)** has ready nginx / Caddy configs.

```bash
metacls api --host 0.0.0.0 --api-key "$(openssl rand -hex 24)"
curl -H "X-API-Key: $KEY" -F files=@leak.pdf http://server:8000/v1/clean
```

## Docker

Pull the published image (built for amd64 + arm64 on every release):

```bash
docker pull ghcr.io/gorkemguler/metacls:latest
```

```bash
# web UI
docker run --rm -p 127.0.0.1:8770:8770 -v "$(pwd)/metacls_cleaned:/data" \
  ghcr.io/gorkemguler/metacls:latest

# REST API
docker run --rm -p 127.0.0.1:8000:8000 -v "$(pwd)/metacls_cleaned:/data" \
  ghcr.io/gorkemguler/metacls:latest api --host 0.0.0.0 --port 8000 --output-dir /data

# one-off: scrub a mounted folder
docker run --rm -v "$(pwd)/docs:/work" ghcr.io/gorkemguler/metacls:latest \
  clean /work --out /work/cleaned
```

Or build locally: `docker build -t metacls .`. `docker compose up --build` runs the web UI;
`docker compose --profile api up metacls-api` the API. Put a proxy in front before exposing
either; see [docs/reverse-proxy.md](https://github.com/gorkemguler/MetaCLS/blob/main/docs/reverse-proxy.md).

## How thorough is it?

- **PDF**: a full QPDF rewrite, not an incremental update, so the removed `/Info` and XMP
  aren't left behind in an old xref section. Annotation authors/dates and embedded-file
  metadata go too; the annotation's visible text and the attached file itself stay.
  **Signed PDFs are skipped, not broken**: scrubbing would invalidate the signature;
  re-export an unsigned copy if you need it cleaned. **Encrypted PDFs** without `--password`
  are skipped; with it, the cleaned copy is written unencrypted (the result says so).
- **Office / ODF**: the metadata parts are deleted from the package (or, for ODF, emptied),
  not merely blanked, and the references to them are pruned so nothing dangles.
- **Images**: `exiftool -all=`, which is the reference tool for this.
- **`--verify`** (on by default) re-scans every cleaned file and lists anything still present
  in the report; the CLI exits `2` if so.
- **Deterministic**: scrubbing the same file twice with the same options gives byte-identical
  output (checked in CI), so a scrub is auditable. `--strip-pdf-id` is the deliberate exception.

### Limitations

- Content is out of scope by design: text in the document body, a visible/scanned signature,
  text baked into an image. MetaCLS won't touch those. (MetaScout's `--scan-content` finds
  them; removing them is a manual edit.)
- Legacy `.doc / .xls / .ppt` are not scrubbed (convert first).
- Not a certified sanitisation tool. Verify anything high-stakes yourself: that's what
  `metacls inspect` on the output, or a second pass with MetaScout, is for.

## Desktop integration

All in **[`platform/`](https://github.com/gorkemguler/MetaCLS/tree/main/platform)**, one install command each; everything scrubs in place:

- **macOS**: a drag-and-drop `MetaCLS.app` (built with `osacompile`, no Xcode), plus a
  Finder **Quick Action**.
- **Windows**: a WinForms **drop window**, a **Send to** menu entry, an Explorer
  **right-click** entry, and a `winget` manifest (template).
- **Linux**: a `.desktop` launcher / *Open With* handler (with a `zenity` picker), a
  Nautilus/Nemo/Caja **script**, and a `systemd` **watch** unit.

## CI / hooks

`metacls clean --check` implies `--dry-run` and exits **3** if any file still carries
metadata (0 if clean, 1 on error), a gate for pre-commit and CI.

```yaml
# .pre-commit-config.yaml
- repo: https://github.com/gorkemguler/MetaCLS
  rev: main
  hooks: [{ id: metacls }]
```

```yaml
# .github/workflows/no-metadata.yml
- uses: gorkemguler/MetaCLS@main
  with:
    paths: docs/ public/
    mode: check        # or "fix" to scrub in place and commit
```

## Roadmap

See **[ROADMAP.md](https://github.com/gorkemguler/MetaCLS/blob/main/ROADMAP.md)** for what's shipped, the known limitations, and the backlog
(inotify watch, SFTP mode, `--jobs`, `--quarantine`, `.metacls.toml`, policy profiles,
audio/video, PyPI, recursive-container scrubbing).

## License

MIT, see [LICENSE](https://github.com/gorkemguler/MetaCLS/blob/main/LICENSE).
