Metadata-Version: 2.4
Name: bytewise
Version: 0.11.0
Summary: Standalone neural MIME detection from raw bytes
Author-email: Chris Mattmann <chris.mattmann@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/chrismattmann/bytewise
Project-URL: Issues, https://github.com/chrismattmann/bytewise/issues
Project-URL: Repository, https://github.com/chrismattmann/bytewise
Keywords: mime,file-type,neural-network,transformer,byte-analysis
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database :: Front-Ends
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: numpy>=1.26
Provides-Extra: inference
Requires-Dist: tensorflow<2.19,>=2.18; extra == "inference"
Provides-Extra: cuda
Requires-Dist: tensorflow[and-cuda]<2.19,>=2.18; extra == "cuda"
Provides-Extra: metal
Requires-Dist: tensorflow<2.19,>=2.18; extra == "metal"
Requires-Dist: tensorflow-metal<1.3,>=1.2; extra == "metal"
Provides-Extra: research
Requires-Dist: beautifulsoup4>=4.13.3; extra == "research"
Requires-Dist: cbor2>=5.6; extra == "research"
Requires-Dist: olefile>=0.47; extra == "research"
Requires-Dist: requests; extra == "research"
Requires-Dist: tomli>=2; python_version < "3.11" and extra == "research"
Requires-Dist: warcio>=1.7.5; extra == "research"
Dynamic: license-file

# Bytewise MIME detector

[![CI: passing](https://img.shields.io/badge/CI-passing-brightgreen.svg)](https://github.com/chrismattmann/bytewise/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE.txt)
[![PyPI version](https://img.shields.io/pypi/v/bytewise.svg?cacheSeconds=300&release=0.11.0)](https://pypi.org/project/bytewise/)

Bytewise is a standalone neural MIME detector trained on raw file bytes. It
does not require Java, a Tika server, a filename, or a file extension. The
repository preserves its complete research lineage: BFA/BFC baselines, neural
experiments, strict-host validation, deduplication audits, and D3 reports.

The Bytewise 0.11.0 default is the hash-pinned, source-disjointly validated
Bytewise-175 rehearsal-refined model with validation-only DOC calibration. It learns 175 MIME classes and exposes 177
supported outputs, including octet-stream fallback and M4V refinement. The
prior 151-, 150-, 125-, 100-, 58-, and original 57-class models remain bundled as explicit
rollback and reproducibility options.

## Usage

Homebrew Python is an externally managed environment and must not be modified
with `pip --break-system-packages`. For the command-line application, install
Bytewise into an isolated Python 3.12 environment with `uv`:

```bash
uv tool install --python 3.12 "bytewise[metal]"
uv tool update-shell
```

Open a new terminal (or add `$HOME/.local/bin` to `PATH`) and verify it:

```bash
bytewise --version
bytewise model-info
bytewise doctor
```

Choose the runtime extra for the target platform:

```bash
# Apple Silicon GPU
python -m pip install "bytewise[metal]"

# Linux with an NVIDIA GPU and a current NVIDIA driver
python -m pip install "bytewise[cuda]"

# CPU inference on Linux or Windows
python -m pip install "bytewise[inference]"
```

The `metal` extra pins the validated TensorFlow 2.18 and `tensorflow-metal`
1.2 runtime. The `cuda` extra installs TensorFlow's pip-managed CUDA and cuDNN
libraries; the host still needs a compatible NVIDIA driver. Confirm CUDA is
visible with:

```bash
python -c 'import tensorflow as tf; print(tf.config.list_physical_devices("GPU"))'
```

For development or library use, keep the dependency in a project environment:

```bash
cd "$HOME/git/bytewise"
uv sync --python 3.12 --extra metal --group tests
uv run bytewise model-info
uv run python
```

Inside that `uv run python` session:

```python
from bytewise import Detector

detector = Detector.load_default()
result = detector.detect_file("document.bin")

print(result.mime_type)
print(result.confidence)
print(result.alternatives)
```

The same model is available from the command line:

```bash
bytewise detect document.bin image.dat
bytewise detect document.bin --top-k 5 --threshold 0.80 --json
cat unknown.bin | bytewise detect -
bytewise supported
bytewise model-info
```

Select an earlier immutable model when reproducibility requires it:

```python
legacy = Detector.load_default(model="legacy")
previous = Detector.load_default(model="previous")
production58 = Detector.load_default(model="production58")
production125_v2 = Detector.load_default(model="production125_v2")
java_router = Detector.load_default(model="java_router")
production175 = Detector.load_default(model="production175")
```

```bash
bytewise detect --model-version legacy document.bin
bytewise detect --model-version previous document.bin
bytewise detect --model-version production58 document.bin
bytewise detect --model-version production125_v2 document.bin
bytewise detect --model-version java_router Example.java
bytewise detect --model-version production175 document.bin
bytewise model-info --model-version legacy
```

Routine TensorFlow startup diagnostics are suppressed so CLI output remains
script-friendly. To diagnose device selection or CUDA loading, enable them for
one invocation:

```bash
bytewise doctor --require-gpu
bytewise detect --tensorflow-logs document.bin
```

Allow uncertain inputs to abstain so another detector can handle them:

```python
detector = Detector.load_default(confidence_threshold=0.80)
result = detector.detect_bytes(payload)
if result.abstained:
    # Fall back to tika-python, libmagic, or another detector.
    pass
```

## Production 175-class rehearsal-refined model

The Bytewise 0.11.0 default is `bytewise-011-rehearsal-calibrated-v1-candidate`,
the exact passing rehearsal checkpoint with a validation-selected `+0.10`
`application/msword` logit bias:

- Permanent-suite accuracy: 91.65%, up 0.27 points over Bytewise 0.10.0
- Permanent-suite macro F1: 82.42%, up 0.24 points
- Fresh six-class external accuracy: 50.62%, up 30.03 points
- Fresh six-class external macro F1: 64.10%, up 27.64 points
- Optional validation-derived DOC rejection threshold: 0.40
- Learned labels: 175; supported outputs: 177
- Model SHA-256: `90b70a8c6d4eda30299d4b2a13de3960216aa9c326e49d713e56e51363d882fc`

The Bytewise 0.10.0 repaired model remains selectable with
`--model-version production175`.

## Preserved 151-class Java router

The Bytewise 0.9.0 model is `bytewise-151-java-router-v1`. It preserves the
0.8.0 result unless Java neural confidence and structural Java syntax both pass
the locked routing contract. Independent evaluation achieved 98.0% Java recall
and 98.99% Java F1 with zero changed predictions on the established Wikimedia
and complementary regression suites.

The equivalent explicit alias is `--model-version java_router`. The Bytewise
0.8.0 model remains selectable with `--model-version production150`.

## Preserved 150-class fusion model

The Bytewise 0.8.0 default is `bytewise-150-fusion-v1`. It combines the frozen
150-class v3 model with the Bytewise 0.7.1 125-class parent using the externally
validated 0.4/0.6 probability weights and a 4x expansion-class bias:

- Internal accuracy: 92.35%; macro F1: 0.826
- Internal expansion accuracy: 86.75%; macro F1: 0.892
- Fresh mixed holdout: +1.14 accuracy points overall and +1.01 expansion points
- Expanded legacy confirmation: 85.41% accuracy, +0.54 points over the parent
- Learned labels: 150; supported outputs: 152
- Fusion bundle SHA-256:
  `e2f85dcaf8416b318fc6e01e311e8872d8be7d8b65fcb8b0cfd3a7084fd05e99`

The exact contract, model components, and external validation are documented in
the [Bytewise-150 report](reports/bytewise-150-v1/index.html).

The Bytewise 0.7.1 125-class model remains selectable with
`--model-version production125_v2`.

The Bytewise 0.6.0 125-class model remains selectable with
`--model-version production125_v1`.

The Bytewise 0.5.0 100-class model remains selectable with
`--model-version previous`. The Bytewise 0.3.0 58-class model remains
selectable with `--model-version production58`.

## Legacy v1 model

The preserved legacy model is `polar-byte-transformer-seed550-v1`:

- Validation accuracy: 96.77%
- Independent-reference accuracy: 94.59%
- Input: first 4,096 raw bytes
- Labels: 57 MIME types
- Model SHA-256:
  `72a2f5f2dd0fbb4ffaf88488618bc8e034c03876c7cea94d11da23439ed5b849`

See the [model card](artifacts/polar-byte-transformer-seed550-v1/MODEL_CARD.md)
and [Full-v3 report](reports/full-v3/index.html) for the complete evidence.

## Repository organization

- `src/bytewise/`: production API and migrated byte-frequency research code
- `artifacts/`: immutable model release bundle
- `configs/`, `scripts/`: reproducible experiments and evaluation
- `reports/`: curated D3 reports from early pilots through Full-v3
- `docs/`: research and repository-extraction documentation
- `tests/`: standalone production and research regression tests

Dataset bytes, feature caches, databases, raw predictions, and transient logs
are intentionally kept outside Git.
