Metadata-Version: 2.4
Name: anti-bot-detector
Version: 0.1.0
Summary: Framework-agnostic anti-bot detection for web crawlers: HTTP header fingerprints, JS challenge identification, and proxy auto-rotation. Derived from crawl4ai.
Author: AKN
License: MIT
Project-URL: Homepage, https://github.com/akn/anti-bot-detector
Project-URL: Repository, https://github.com/akn/anti-bot-detector
Project-URL: Changelog, https://github.com/akn/anti-bot-detector/blob/main/CHANGELOG.md
Project-URL: Upstream project (crawl4ai), https://github.com/unclecode/crawl4ai
Keywords: anti-bot,crawler,scraping,cloudflare,recaptcha,proxy-rotation,crawl4ai,waf
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: httpx
Requires-Dist: httpx>=0.24; extra == "httpx"
Provides-Extra: playwright
Requires-Dist: playwright>=1.40; extra == "playwright"
Provides-Extra: scrapy
Requires-Dist: scrapy>=2.10; extra == "scrapy"
Provides-Extra: crawl4ai
Requires-Dist: crawl4ai>=0.4; extra == "crawl4ai"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# anti-bot-detector

[![PyPI - Version](https://img.shields.io/pypi/v/anti-bot-detector.svg)](https://pypi.org/project/anti-bot-detector/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](pyproject.toml)

**Framework-agnostic anti-bot detection for web crawlers** — HTTP header
fingerprinting, JS challenge identification, and proxy auto-rotation in one
dependency-free core. Derived from [crawl4ai](https://github.com/unclecode/crawl4ai)'s
Anti-Bot Detection subsystem (Apache-2.0); zero crawl4ai imports, zero runtime
dependencies.

## Why

Every crawler eventually hits a WAF: Cloudflare challenges, DataDome blocks,
rate-limit pages that look like real content. `anti-bot-detector` classifies
responses **before** your parser wastes time on them, and tells you what to do
next — proceed, rotate proxy, solve challenge, or back off.

- **Zero dependencies** — the detection core is pure standard library
- **Framework-agnostic** — adapters for httpx, scrapy, playwright and crawl4ai
  (all optional extras, lazily imported)
- **3-layer detection** — header fingerprints → JS challenge bodies → proxy
  auto-upgrade state machine
- **Apache-2.0 compliant** — derivation scope documented in [DESIGN.md](DESIGN.md),
  new code MIT-licensed

## Installation

```bash
pip install anti-bot-detector

# with framework adapters (optional extras):
pip install anti-bot-detector[httpx]      # httpx integration
pip install anti-bot-detector[scrapy]     # scrapy downloader middleware
pip install anti-bot-detector[playwright] # playwright integration
pip install anti-bot-detector[dev]        # pytest + build + twine
```

Requires Python ≥ 3.9. Installing the base package does **not** install
crawl4ai, httpx, scrapy or playwright.

## Quick start

Classify a response you already have (no network needed):

```python
from anti_bot_detector import AntiBotEngine, FetchResponse

engine = AntiBotEngine()
result = engine.detect_response(
    FetchResponse(
        url="https://example.com",
        status_code=503,
        headers={"cf-mitigated": "challenge", "server": "cloudflare"},
        body="<html>Just a moment...</html>",
    )
)
print(result.summary())
# https://example.com -> level=blocked challenge=cloudflare
#   confidence=0.90 action=rotate_proxy
```

Fetch + detect + auto-upgrade in one call:

```python
from anti_bot_detector import AntiBotEngine

def fetch(url: str, headers: dict):
    import httpx  # any HTTP client works
    resp = httpx.get(url, headers=headers, timeout=10, follow_redirects=True)
    from anti_bot_detector import FetchResponse
    return FetchResponse(
        url=str(resp.url),
        status_code=resp.status_code,
        headers=dict(resp.headers),
        body=resp.text,
    )

engine = AntiBotEngine(
    fetcher=fetch,
    proxies=["http://proxy1:8080", "http://proxy2:8080"],
    max_retries=3,
)
result = engine.detect("https://target.example.com")
if result.recommended_action.value == "proceed":
    print("safe to crawl", result.url)
```

## API reference

### Core (`anti_bot_detector`)

| Object | Purpose |
|--------|---------|
| `AntiBotEngine` | Unified entry point orchestrating all 3 layers |
| `FetchResponse` | Minimal, framework-agnostic view of an HTTP response |
| `DetectionResult` | Outcome of the pipeline: `bot_level`, `challenge_type`, `confidence`, `recommended_action`, `summary()` |
| `BotLevel` | `NONE` / `FINGERPRINT` / `CHALLENGE` / `BLOCKED` |
| `ChallengeType` | `NONE` / `CLOUDFLARE` / `RECAPTCHA` / `HCAPTCHA` / `PERIMETERX` / `DATADOME` / `AKAMAI` / `CUSTOM` |
| `RecommendedAction` | `PROCEED` / `ROTATE_PROXY` / `SOLVE_CHALLENGE` / `BACKOFF` / `ABORT` |
| `FingerprintGenerator` | Layer 1 — coherent browser-like header sets (UA pool, Sec-CH-UA, Sec-Fetch-*) |
| `ProxyRotator` | Layer 3 — round-robin rotation, per-proxy cooldown, exhaustion fallback |
| `StaticProxyProvider` | Trivial `ProxyProvider` backed by a fixed list |

Key methods:

```python
engine.detect_response(response: FetchResponse) -> DetectionResult  # layers 1+2
engine.detect(url: str) -> DetectionResult                          # fetch + layers 1-3
engine.fetch(url: str) -> DetectionResult                           # with auto-upgrade

rotator.current()                                  # active proxy or None (exhausted)
rotator.rotate(reason="soft-block")                # upgrade to next healthy proxy
rotator.mark_failed(proxy, reason="timeout")       # cooldown + rotate away
```

### Adapters (`anti_bot_detector.adapters`)

Adapters are lazily imported — importing the package never pulls a framework.

| Adapter | Reduces | Extra |
|---------|---------|-------|
| `HttpxAdapter` | `httpx.Response` → `FetchResponse` | `[httpx]` |
| `PlaywrightAdapter` | playwright `Response` → `FetchResponse` | `[playwright]` |
| `Crawl4AIAdapter` | crawl4ai `AsyncWebCrawler` result → `FetchResponse` | `[crawl4ai]` |
| `AntiBotMiddleware` | scrapy downloader middleware (drops blocked responses) | `[scrapy]` |

## Integration examples

### httpx client

```python
import httpx
from anti_bot_detector import BotLevel
from anti_bot_detector.adapters import HttpxAdapter

def detect_url(url: str):
    with httpx.Client() as client:
        response = client.get(url, follow_redirects=True)
    verdict = HttpxAdapter().detect(response)
    if verdict.bot_level is BotLevel.BLOCKED:
        raise RuntimeError(f"WAF block: {verdict.summary()}")
    return verdict
```

### scrapy downloader middleware

```python
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "anti_bot_detector.adapters.scrapy_middleware.AntiBotMiddleware": 560,
}
ANTIBOT_DROP_CHALLENGES = True  # also drop "challenge" pages, not just blocks
```

Blocked responses raise before they reach your spider callback, so parsers
never see WAF error pages.

More runnable scripts in [`examples/`](examples/):
`standalone_detect.py` (pure Python, stdlib only) and
`httpx_integration.py` (httpx client integration).

## License

New code: MIT (see [LICENSE](LICENSE)). Derived from crawl4ai's Anti-Bot
Detection subsystem, Apache-2.0 — see [NOTICE](NOTICE) and
[LICENSES/](LICENSES/) for the upstream license and the derivation scope.
