Metadata-Version: 2.5
Name: fast-web-md-extract
Version: 0.1.0
Summary: Extract web content as Markdown — fast HTTP with anti-bot browser fallback
Project-URL: Homepage, https://github.com/onlygummy/fast-web-md-extract
Project-URL: Repository, https://github.com/onlygummy/fast-web-md-extract
Project-URL: Issues, https://github.com/onlygummy/fast-web-md-extract/issues
Project-URL: Changelog, https://github.com/onlygummy/fast-web-md-extract/blob/main/CHANGELOG.md
Author: onlygummy
License-Expression: MIT
License-File: LICENSE
Keywords: anti-bot,cloudflare,markdown,scraping,web
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: camoufox[geoip]>=0.5.6
Requires-Dist: html-to-markdown>=3.14.3
Requires-Dist: httpx>=0.28.1
Requires-Dist: tqdm>=4.66.0
Requires-Dist: trafilatura>=2.2.0
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Description-Content-Type: text/markdown

# fast-web-md-extract

Extract web page content as clean Markdown. Tries a fast HTTP request first,
falls back to a Camoufox browser for anti-bot-protected sites.

## Install

```bash
pip install fast-web-md-extract
```

## Quick start

### CLI

```bash
# Extract a single URL
fwm https://docs.python.org/3/tutorial/classes.html

# Extract multiple URLs
fwm https://example.com https://docs.python.org

# Save to a specific directory
fwm https://example.com -o my_output

# Extract from a file of URLs
fwm -f urls.txt

# Enable verbose logging
fwm https://example.com -v

# Use fast mode (skip fallback extraction)
fwm https://example.com --fast
```

### Python (Async)

```python
import asyncio
from fast_web_md_extract import Extractor

async def main():
    async with Extractor() as ex:
        result = await ex.extract("https://docs.python.org/3/tutorial/classes.html")
        print(result.markdown[:200])

asyncio.run(main())
```

### Python (Sync)

```python
from fast_web_md_extract import Extractor

with Extractor() as ex:
    result = ex.extract_sync("https://docs.python.org/3/tutorial/classes.html")
    print(result.markdown[:200])
```

## Save to file

```python
# Markdown (default)
result.save("output/page.md")

# Plain text (strips Markdown syntax)
result.save("output/page.txt", fmt="txt")
```

## Batch extract

```python
async with Extractor() as ex:
    results = await ex.extract_many([
        "https://example.com",
        "https://docs.python.org",
    ])
    for r in results:
        if r.markdown:
            r.save(f"output/{r.url.split('/')[2]}.md")
```

## Configuration

```python
Extractor(
    headless=True,       # Run browser headless (default True)
    timeout=15,          # Cloudflare challenge timeout in seconds
    min_content=1000,    # Min chars to accept from fast path
    concurrency=3,       # Max concurrent browser pages
    http_concurrency=50, # Max concurrent HTTP requests
    fast=False,          # Use trafilatura fast mode
)
```

## How it works

1. **Fast path** (httpx) — plain HTTP GET, no browser needed. Works for
   static sites like Wikipedia, Python Docs, blogs. Uses connection pooling
   and high concurrency (50 concurrent requests).
2. **Slow path** (Camoufox) — launches a stealth browser to bypass
   Cloudflare and other anti-bot protection. Automatic fallback when the
   fast path is blocked or returns too little content.
3. **Content extraction** (trafilatura + html-to-markdown) — trafilatura
   auto-detects main content first. Falls back to html-to-markdown for
   pages where trafilatura strips links or returns too little content.

## Benchmark

Measured on Windows, Python 3.14, September 2026:

| Site | Type | Time | Chars | Method |
|------|------|------|-------|--------|
| docs.python.org | Static | 0.7s | 36,732 | HTTP fast path |
| th.wikipedia.org | Static | 4.1s | 293,464 | HTTP fast path |
| bbc.com | JS-rendered | 49.7s | 43,445 | Browser fallback |
| fao.org (Cloudflare) | Protected | 57.7s | 29,602 | Browser + html-to-markdown |

**Static sites: ~1-4s** | **Protected sites: ~20-60s**

### CLI benchmark

```bash
$ fwm https://docs.python.org/3/tutorial/classes.html https://th.wikipedia.org/wiki/ประเทศไทย
Extracting: 100%|██████████| 2/2 [00:04<00:00, 2.05s/url]
  0.7s  OK: 36,732 chars -> output/docs.python.org.md
  4.1s  OK: 293,464 chars -> output/th.wikipedia.org.md

2/2 extracted successfully.
```

## License

MIT
