Metadata-Version: 2.5
Name: htmlsift
Version: 0.1.0
Summary: Main-content extraction for web pages
Project-URL: Repository, https://github.com/koivualeksi/htmlsift
Author-email: Aleksi Koivu <koivu.aleksi@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: boilerplate-removal,content-extraction,html,main-content,web-scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup :: HTML
Requires-Python: >=3.10
Requires-Dist: huggingface-hub
Requires-Dist: lxml
Requires-Dist: numpy
Requires-Dist: onnxruntime
Requires-Dist: tokenizers
Provides-Extra: full
Requires-Dist: torch; extra == 'full'
Requires-Dist: transformers; extra == 'full'
Description-Content-Type: text/markdown

# htmlsift

Web main-content extraction: HTML in, clean text / HTML / markdown out. A small model,
trained on [WebMainBench](https://github.com/opendatalab/WebMainBench), scores each rendered
line of a page as main content or boilerplate (nav, ads, cookie banners, and footers) and
returns just the content.

Apache-2.0. Full documentation, diagrams, and benchmarks:
https://github.com/koivualeksi/htmlsift

## Install

```bash
pip install htmlsift          # default: the mini model (CPU, ONNX)
pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
```

- `htmlsift` — the `mini` model only; a small install (onnxruntime + tokenizers + lxml +
  numpy + huggingface-hub). First use downloads ~230 MB of model files.
- `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
  downloads ~1 GB.

Requires Python 3.10+. Model files download from the Hugging Face Hub on first use and are
cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
cache only.

## Usage

```python
from htmlsift import Extractor

ex = Extractor()                     # defaults to "mini"
ex.extract(html)                     # -> str (plain text)
ex.extract(html, output="html")      # structure, links, images kept
ex.extract(html, output="markdown")  # clean markdown: [text](url), ![alt](src)
ex.extract(html, with_blocks=True)   # -> (blocks, probs)

import htmlsift
htmlsift.extract(html)               # module-level convenience
```

The larger model, from `htmlsift[full]`:

```python
ex = Extractor("base")               # 311M encoder; GPU when present, else CPU (with a warning)
ex = Extractor("base", device="cuda")  # require the GPU
```

`mini` is CPU-only; only `base` runs on the GPU.

## Output modes

| `output` | what you get |
|---|---|
| `text` (default) | selected content as plain text |
| `html` | selected DOM, with structure / links / images kept |
| `markdown` | clean markdown, links and images preserved |

## License

Apache-2.0. Model weights are distributed separately, from the Hugging Face Hub.
