Metadata-Version: 2.5
Name: dungle-scrubs-scraper
Version: 0.2.1
Summary: Web scraping service with anti-bot detection and Jina fallback
Author: Kevin Frilot
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: crawl4ai>=0.8.9
Requires-Dist: fastapi>=0.136.3
Requires-Dist: httpx>=0.27.0
Requires-Dist: lxml>=6.1.1
Requires-Dist: nltk>=3.10.0
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pydantic>=2.13.4
Requires-Dist: starlette>=1.3.0
Requires-Dist: uvicorn>=0.49.0
Description-Content-Type: text/markdown

# scraper

Web scraping service with anti-bot detection and Jina fallback. Runs as a
standalone service or installs as a Python package.

## Install

```bash
pip install dungle-scrubs-scraper
```

## API

```
POST /scrape {"url": "https://example.com"}
GET  /health
```

### Response

```json
{
  "markdown": "# Page Title\n\nClean content...\n",
  "source": "crawl4ai",
  "url": "https://example.com",
  "bot_detected": false,
  "error": null,
  "elapsed_ms": 1234.5,
  "attempts": ["crawl4ai attempt 1"]
}
```

## Pipeline

1. **Crawl4AI** — local Playwright headless Chromium, retries on transient
   errors
2. **Bot detection** — Cloudflare/CAPTCHA phrases + partial JS render
   heuristic (empty table cells)
3. **Jina Reader fallback** — free cloud re-fetch when bot-blocked or crawl
   fails
4. **Artifact stripping** — regex patterns for "Skip to content", GitHub
   chrome, duplicate lines

## CLI

The `scraper` command exposes two subcommands:

```bash
scraper scrape <url>       # scrape a URL, JSON on stdout
scraper scrape --stdin     # read {"url": "..."} from stdin
scraper serve              # start the HTTP server
```

### JSON output (stdout)

Every `scrape` invocation writes exactly one JSON object to stdout:

```json
{
  "ok": true,
  "markdown": "# Page Title\n\nClean content...\n",
  "source": "crawl4ai",
  "url": "https://example.com",
  "bot_detected": false,
  "error": null,
  "elapsed_ms": 1234.5,
  "attempts": ["crawl4ai attempt 1"]
}
```

On failure, `ok` is `false` and `error` contains the reason. The structure
is always the same — callers can unconditionally `json.loads(stdout)`.

### Exit codes

| Code | Meaning |
|------|---------|
| 0 | Success — content scraped |
| 1 | Scrape failed — all backends exhausted |
| 2 | Invalid input — bad URL or malformed stdin JSON |
| 3 | Timeout — global deadline exceeded |
| 4 | Internal error — unexpected crash |

### Timeout

The `--timeout` flag (default: 120s) wraps the entire pipeline in a
global deadline. When exceeded, the process exits with code 3 and the
JSON `error` field explains what happened.

```bash
scraper scrape --timeout 30 https://slow-site.example.com
```

### stdin mode

Pipe a JSON object with a `url` field:

```bash
echo '{"url": "https://example.com"}' | scraper scrape --stdin
```

All logging goes to stderr. stdout is exclusively for the JSON result.

## Run locally

```bash
uv sync
uv run scraper serve
```

## Run with Docker

```bash
docker compose up -d
```

## Environment

| Variable | Required | Description |
|----------|----------|-------------|
| `SCRAPER_PORT` | No | Server port (default: 8020) |
| `SCRAPER_HOST` | No | Bind address (default: `127.0.0.1`, local only; set `0.0.0.0` to expose) |
| `SCRAPER_API_KEY` | No | When set, `POST /scrape` requires a matching `Authorization: Bearer` or `X-API-Key` header |
| `SCRAPER_JINA_ENABLED` | No | Enable the third-party Jina Reader fallback (default: `false`) |
| `SCRAPER_SERVER_TIMEOUT_SECONDS` | No | Global per-request deadline for `/scrape` (default: 120) |
| `SCRAPER_MAX_CONCURRENT_SCRAPES` | No | Max concurrent scrapes before `503` (default: 4) |
| `SCRAPER_CRAWL_TIMEOUT_SECONDS` | No | Crawl4AI timeout (default: 25) |

## Security

This service fetches **arbitrary, caller-supplied URLs** and treats every
inbound URL and every fetched page as untrusted. Defaults are tuned for a
**local, single-user** tool:

- **SSRF guard.** Every URL — on the crawl4ai path, the Jina path, the CLI, and
  every redirect hop — is validated by a single guard that allows only
  `http(s)` and rejects hosts resolving to loopback, private, link-local,
  reserved, multicast, CGNAT, or cloud-metadata addresses (e.g.
  `127.0.0.1`, `169.254.169.254`, `10.0.0.0/8`). Credentials in
  `user:pass@host` URLs are stripped before any request.
- **Local by default.** The server binds `127.0.0.1`. Set `SCRAPER_HOST=0.0.0.0`
  to expose it — and set `SCRAPER_API_KEY` when you do.
- **Optional auth.** With `SCRAPER_API_KEY` set, `POST /scrape` requires a
  matching `Authorization: Bearer <key>` or `X-API-Key: <key>` header.
- **Resource bounds.** A global per-request deadline, a concurrency cap
  (`503` when exceeded), and response/markdown size caps protect against
  hangs and memory exhaustion.
- **Third-party egress is opt-in.** The Jina Reader fallback is disabled by
  default (`SCRAPER_JINA_ENABLED=false`) because it sends the target URL to a
  third party (`r.jina.ai`).

Report vulnerabilities privately — see [SECURITY.md](SECURITY.md).

## Use as a library

```python
from scraper.scraper import scrape

result = await scrape("https://example.com")
print(result.markdown)
```

## Use as a service

All consumers need one env var:

```bash
SCRAPER_URL=http://localhost:8020
```

```python
import httpx

resp = httpx.post(f"{SCRAPER_URL}/scrape", json={"url": url})
data = resp.json()
markdown = data["markdown"]
```

## Tests

```bash
uv run pytest tests/ -v
```

All mocked - no network required.

## Port

8020 (registered in the shared port table).

## License

MIT
