Metadata-Version: 2.4
Name: html2md-clean
Version: 0.1.1
Summary: A clean HTML -> Markdown converter with noise removal (ads, popups, navigation) for web scraping and LLM pipelines.
Author: 0-EternalJunior-0
License-Expression: MIT
Project-URL: Homepage, https://github.com/0-EternalJunior-0/html2md-clean
Project-URL: Repository, https://github.com/0-EternalJunior-0/html2md-clean
Project-URL: Issues, https://github.com/0-EternalJunior-0/html2md-clean/issues
Keywords: html,markdown,html2md,html-to-markdown,web-scraping,boilerplate-removal,content-extraction,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2
Requires-Dist: pydantic-settings==2.14.1
Requires-Dist: beautifulsoup4
Requires-Dist: lxml
Requires-Dist: selectolax
Requires-Dist: requests
Provides-Extra: full
Requires-Dist: trafilatura; extra == "full"
Requires-Dist: ftfy; extra == "full"
Requires-Dist: courlan; extra == "full"
Requires-Dist: htmldate; extra == "full"
Requires-Dist: nh3; extra == "full"
Provides-Extra: dev
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mkdocs; extra == "dev"
Requires-Dist: mkdocs-material; extra == "dev"
Dynamic: license-file

# html2md-clean

> A clean, controllable **HTML -> Markdown** converter with noise removal (ads, popups, navigation), built for web scraping and preparing text for LLMs.

[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
[![Ruff](https://img.shields.io/badge/lint-ruff-46a2f1.svg)](https://docs.astral.sh/ruff/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

---

## Table of contents

- [Overview](#overview)
- [Features](#features)
- [Installation](#installation)
- [Quick start](#quick-start)
- [CLI](#cli)
- [Usage examples](#usage-examples)
- [Public API](#public-api)
- [Conversion options](#conversion-options)
- [Optional dependencies](#optional-dependencies)
- [Performance](#performance)
- [Development](#development)
- [Documentation](#documentation)
- [License](#license)

---

## Overview

`html2md-clean` turns the "messy" HTML of real-world web pages into clean Markdown.
Unlike naive converters, the library first **strips boilerplate** — navigation,
footers, banners, cookie popups, ad blocks — and only then renders the main
content. That makes it a good fit for:

- **web scraping** and dataset building;
- preparing text for **RAG / LLM** pipelines (fewer noise tokens);
- **archiving** articles as readable Markdown.

The library runs on two parsers: **Selectolax** (fast, the default when
installed) and **BeautifulSoup** (fallback), hidden behind a single `Adapter`
interface.

## Features

- **Noise removal** by tag, CSS class, `id`, `aria-hidden`, and inline `display:none`.
- **Main-content extraction** via link-density and content-density heuristics.
- **Links**: inline `[text](url)` or reference-style `[text][N]`, with optional
  UTM/tracker stripping (courlan).
- **Citations**: a Markdown variant with footnotes `[1]`, `[2]` plus a reference list.
- **Metadata**: `title`, first `h1`, `description`, and counters for words / characters / headings / links.
- **Publication / update dates** (optional, via `htmldate`).
- **Mojibake repair** (`ftfy`), NFKC Unicode normalization, HTML sanitization (`nh3`).
- **Synchronous and asynchronous** APIs, with a lazily generated `raw_markdown`.
- **Immutable options** (`MarkdownOptions.with_overrides(...)`) — thread-safe.

## Installation

Requires **Python 3.11+**.

```bash
pip install html2md-clean
```

With optional extras (dates, sanitization, encoding fixes, URL cleaning):

```bash
pip install "html2md-clean[full]"
```

Install straight from the github repository:

```bash
pip install git+https://github.com/0-EternalJunior-0/html2md-clean.git
```

## Quick start

```python
from markdown import MarkdownGenerator

html = """
<html><body>
  <nav>site menu</nav>
  <article>
    <h1>Article title</h1>
    <p>This is the main paragraph with useful text.</p>
  </article>
  <footer>&copy; 2026</footer>
</body></html>
"""

generator = MarkdownGenerator()
result = generator.generate_from_html(html)

print(result.fit_markdown)
# # Article title
#
# This is the main paragraph with useful text.

print(result.word_count, result.h1)
```

The shortest variant:

```python
from markdown import MarkdownGenerator

result = MarkdownGenerator.quick_generate(html)
print(result.markdown)  # alias for fit_markdown
```

## CLI

Once installed, the `html2md` command is available:

```bash
# from a file to stdout
html2md page.html

# from stdin to a file
cat page.html | html2md -o page.md

# keep links (disabled by default in the CLI profile)
html2md page.html --include-links

# force BeautifulSoup instead of Selectolax
html2md page.html --force-bs4
```

You can also run it as a module without installing:

```bash
python main_service.py page.html -o page.md
```

## Usage examples

### Custom options

```python
from markdown import MarkdownGenerator, MarkdownOptions

options = MarkdownOptions(
    include_links=True,
    include_images=True,
    generate_citations=True,
    max_length=50_000,
    clean_urls=True,          # strip utm_*, fbclid, gclid
    base_url="https://example.com",
)

generator = MarkdownGenerator(options)
result = generator.generate_from_html(html)

print(result.markdown_with_citations)
for ref in result.references:
    print(ref)
```

### Reference-style links

```python
from markdown import MarkdownGenerator, MarkdownOptions
from markdown.options import LinkStyle

options = MarkdownOptions(link_style=LinkStyle.REFERENCE)
result = MarkdownGenerator(options).generate_from_html(html)
```

### Asynchronous call

```python
import asyncio
from markdown import generate_markdown_async

async def run():
    result = await generate_markdown_async(html)
    print(result.fit_markdown)

asyncio.run(run())
```

### Extracting publication dates (optional)

```python
from markdown import MarkdownGenerator, MarkdownOptions

options = MarkdownOptions(extract_dates=True)
result = MarkdownGenerator(options).generate_from_html(html)
print(result.published_date, result.updated_date)
```

### Web-scraping facade

```python
from main_service import HTMLToMarkdownConverter

converter = HTMLToMarkdownConverter()   # more aggressive scraping defaults
markdown = converter.convert(html)
```

### Real-world example: configuring `MarkdownOptions` for scraping a blog

Below is how you would do it "for real": take the HTML of a news/blog page,
deliberately enable/disable the options you need, and understand **why** each one
matters.

```python
from markdown import MarkdownGenerator, MarkdownOptions
from markdown.options import LinkStyle, AGGRESSIVE_NOISE_CLASSES, COMMON_NOISE_PATTERNS

options = MarkdownOptions(
    # --- main-content extraction ---
    use_content_density=True,       # look for <article>/<main>, not the whole <body>
    link_density_threshold=0.4,     # discard menus/footers more strictly (more links = noise)
    min_content_words=40,           # short blocks are not treated as main content

    # --- noise removal ---
    remove_ads=True,
    noise_classes=AGGRESSIVE_NOISE_CLASSES,   # + sidebar/related/comments/carousel
    remove_display_none=True,                 # drop hidden inline blocks
    dynamic_noise_patterns=COMMON_NOISE_PATTERNS,  # "5 min ago", "Apply now", counters

    # --- what to keep in the Markdown ---
    include_links=True,
    include_images=True,
    include_tables=True,
    generate_citations=True,        # produce a variant with footnotes [1], [2]

    # --- links and URLs ---
    link_style=LinkStyle.REFERENCE, # cleaner than inline in long articles
    normalize_urls=True,
    base_url="https://example-blog.com",  # relative /article -> absolute
    clean_urls=True,                # strip utm_*, fbclid, gclid

    # --- text normalization ---
    fix_text_encoding=True,         # repair mojibake (requires ftfy)
    normalize_unicode=True,         # fullwidth/ligatures -> ASCII (NFKC)
    max_length=200_000,             # don't truncate long-reads prematurely
)

generator = MarkdownGenerator(options)
result = generator.generate_from_html(html)

print(result.markdown_with_citations)   # text with [1], [2]
print("words:", result.word_count, "| links:", result.rendered_link_count)
for ref in result.references:
    print(ref)   # {'id': 1, 'text': '...', 'url': '...'}
```

> **Tip.** Options are immutable, so keep one "base" profile and make targeted
> variations with `with_overrides` — no need to duplicate the whole set:
>
> ```python
> BASE = MarkdownOptions(use_content_density=True, include_links=True)
>
> # for e-commerce cards: a shorter candidate-length threshold
> ecommerce = BASE.with_overrides(main_content_min_length=100, include_images=True)
>
> # for a "raw" dump without link filtering
> raw_dump = BASE.with_overrides(include_links=False, use_content_density=False)
> ```

## Public API

Import: `from markdown import ...`

| Object                                        | Purpose                                                                                                       |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `MarkdownGenerator`                           | The main generator. `generate_from_html(html) -> MarkdownResult`, `quick_generate(html)`, `generate(source)`. |
| `MarkdownOptions`                             | Immutable conversion config. Use `with_overrides(**kwargs)`.                                                  |
| `MarkdownResult`                              | Result: `fit_markdown`, `raw_markdown` (lazy), `markdown_with_citations`, metadata and counters.              |
| `generate_markdown_async(html, options=None)` | Async wrapper (thread executor).                                                                              |
| `configure_logging(level=WARNING)`            | Configure the package logger.                                                                                 |
| `COMMON_NOISE_PATTERNS`                       | A ready-made set of regexes for `dynamic_noise_patterns`.                                                     |

Key `MarkdownResult` fields:

| Field                                                                    | Type          | Description                                                       |
| ------------------------------------------------------------------------ | ------------- | ----------------------------------------------------------------- |
| `fit_markdown` / `markdown`                                              | `str`         | The main cleaned Markdown.                                        |
| `raw_markdown`                                                           | `str`         | The full Markdown without aggressive filtering (lazy generation). |
| `markdown_with_citations`                                                | `str`         | Variant with footnotes `[1]`, `[2]`.                              |
| `references`                                                             | `list[dict]`  | List of sources for the citations.                                |
| `title`, `h1`, `description`                                             | `str`         | Page metadata.                                                    |
| `word_count`, `char_count`, `heading_count`, `link_count`, `image_count` | `int`         | Statistics.                                                       |
| `published_date`, `updated_date`                                         | `str \| None` | Dates (only with `extract_dates=True`).                           |
| `is_truncated`                                                           | `bool`        | Whether the text was truncated to `max_length`.                   |

## Conversion options

`MarkdownOptions` is the single source of truth for defaults. The most common:

| Option                                                                             | Default       | Description                                          |
| ---------------------------------------------------------------------------------- | ------------- | ---------------------------------------------------- |
| `remove_nav` / `remove_header` / `remove_footer` / `remove_aside`                  | `True`        | Remove the corresponding semantic blocks.            |
| `remove_ads`                                                                       | `True`        | Remove by `noise_classes` / `noise_ids`.             |
| `remove_hidden`                                                                    | `True`        | Strip `aria-hidden`, HTML5 `hidden`, hidden classes. |
| `include_links`                                                                    | `True`        | Render links.                                        |
| `include_images`                                                                   | `False`       | Render images.                                       |
| `include_tables` / `include_lists` / `include_code_blocks` / `include_blockquotes` | `True`        | Support the corresponding elements.                  |
| `max_length`                                                                       | `100000`      | Maximum text length.                                 |
| `min_paragraph_length`                                                             | `3`           | Minimum paragraph length.                            |
| `generate_citations`                                                               | `False`       | Generate `markdown_with_citations`.                  |
| `link_style`                                                                       | `INLINE`      | `INLINE` or `REFERENCE`.                             |
| `clean_urls`                                                                       | `False`       | Strip UTM/tracker parameters (courlan).              |
| `normalize_urls` / `base_url`                                                      | `True` / `""` | Absolutize relative URLs.                            |
| `use_content_density`                                                              | `False`       | Main-content extraction heuristic.                   |
| `link_density_threshold`                                                           | `0.5`         | Link-density threshold for discarding menus/footers. |
| `force_beautifulsoup`                                                              | `False`       | Force BS4 instead of Selectolax.                     |

The full list is in [`markdown/options.py`](markdown/options.py) and in the [documentation](#documentation).

Options are immutable: create new variants with `with_overrides`:

```python
base = MarkdownOptions()
scraping = base.with_overrides(include_links=False, max_length=200_000)
```

## Optional dependencies

Some features are enabled by installing extra packages (the library degrades
gracefully if they are missing):

| Feature                | Option                       | Package       |
| ---------------------- | ---------------------------- | ------------- |
| Date extraction        | `extract_dates`              | `htmldate`    |
| HTML sanitization      | `sanitize_html`              | `nh3`         |
| Mojibake repair        | `fix_text_encoding`          | `ftfy`        |
| URL cleaning           | `clean_urls`                 | `courlan`     |
| Boilerplate extraction | `use_trafilatura_extraction` | `trafilatura` |
| Fast parser            | — (auto)                     | `selectolax`  |

## Performance

- **Selectolax** is used automatically when installed and is significantly faster than BeautifulSoup.
- `raw_markdown` is generated **lazily** — you pay for it only when you access it.
- `MarkdownOptions` is immutable, so generator instances can safely be shared across threads.
