Metadata-Version: 2.4
Name: trace-scraper
Version: 1.0.0
Summary: High-performance async web scraper with Playwright and httpx. Scrape JavaScript-rendered pages, crawl websites recursively, parse sitemaps, and extract content with anti-bot bypass.
Project-URL: Homepage, https://github.com/boxed-dev/trace
Project-URL: Documentation, https://github.com/boxed-dev/trace#readme
Project-URL: Repository, https://github.com/boxed-dev/trace
Project-URL: Issues, https://github.com/boxed-dev/trace/issues
Project-URL: Changelog, https://github.com/boxed-dev/trace/releases
Author-email: Rishabh <rishabh@lumiora.co>
License-Expression: MIT
License-File: LICENSE
Keywords: anti-bot,async,asyncio,beautifulsoup,content-extraction,crawler,data-extraction,headless-browser,html-parser,httpx,javascript-rendering,pdf-extraction,playwright,rate-limiting,scraper,scraping,sitemap,sitemap-parser,spider,web-crawler,web-scraper,web-scraping
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Internet :: WWW/HTTP :: Site Management :: Link Checking
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: lxml>=4.9.0
Requires-Dist: playwright>=1.40.0
Provides-Extra: all
Requires-Dist: pypdf>=3.17.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: mypy>=1.7.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
Requires-Dist: pytest>=7.4.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: pdf
Requires-Dist: pypdf>=3.17.0; extra == 'pdf'
Description-Content-Type: text/markdown

# Trace Scraper

[![PyPI version](https://badge.fury.io/py/trace-scraper.svg)](https://pypi.org/project/trace-scraper/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/boxed-dev/trace/actions/workflows/ci.yml/badge.svg)](https://github.com/boxed-dev/trace/actions/workflows/ci.yml)

**High-performance async web scraper for Python.** Scrape JavaScript-rendered pages, crawl websites recursively, parse sitemaps, and extract clean content — all with built-in anti-bot bypass.

## Why Trace Scraper?

- **Hybrid Engine**: Uses `httpx` for fast static pages, `Playwright` for JavaScript-heavy sites
- **Anti-Bot Bypass**: Built-in detection evasion with realistic browser fingerprints
- **Async & Parallel**: Scrape hundreds of pages concurrently with asyncio
- **Multiple Modes**: Single page, recursive crawl, sitemap parsing, URL lists
- **Clean Output**: Automatic HTML cleanup, content deduplication, PDF extraction
- **Type Safe**: Full type hints with `py.typed` marker

## Installation

```bash
pip install trace-scraper
```

Install Playwright browsers (required for JS rendering):

```bash
playwright install chromium
```

## Quick Start

### Command Line

```bash
# Scrape a single page
trace https://example.com

# Crawl entire website
trace https://docs.example.com --mode recursive --max-pages 100

# Parse sitemap
trace https://example.com/sitemap.xml --mode sitemap

# Use browser for JavaScript sites
trace https://spa-website.com --browser

# Save to JSON
trace https://example.com -o output.json
```

### Python API

```python
import asyncio
from trace_scraper import WebScraper, ScrapeMode

async def main():
    async with WebScraper() as scraper:
        async for doc in scraper.scrape("https://example.com", mode=ScrapeMode.SINGLE):
            print(f"Title: {doc.semantic_identifier}")
            print(f"Content: {doc.sections[0].text}")

asyncio.run(main())
```

### Advanced Configuration

```python
from trace_scraper import WebScraper, ScraperConfig, ScrapeMode

config = ScraperConfig(
    max_concurrent_pages=20,     # Parallel scraping
    page_timeout=15.0,           # Timeout per page
    scroll_pages=True,           # Load lazy content
    extract_pdfs=True,           # Extract PDF text
    use_browser_for_all=False,   # Use httpx when possible
)

async with WebScraper(config) as scraper:
    async for doc in scraper.scrape(
        "https://docs.example.com",
        mode=ScrapeMode.RECURSIVE
    ):
        print(doc.semantic_identifier)
```

## CLI Options

| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--mode` | `-m` | `single` | Mode: `single`, `recursive`, `sitemap` |
| `--output` | `-o` | - | Save results to JSON file |
| `--concurrency` | `-c` | `10` | Max concurrent pages |
| `--timeout` | `-t` | `30` | Page timeout (seconds) |
| `--max-pages` | - | - | Limit pages to scrape |
| `--browser` | - | - | Force browser for all pages |
| `--scroll` | - | - | Scroll to load dynamic content |
| `--delay` | `-d` | `0.1` | Delay between requests |
| `--verbose` | `-v` | - | Enable debug logging |

## Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `max_concurrent_pages` | `int` | `10` | Pages to scrape in parallel |
| `max_concurrent_browsers` | `int` | `3` | Browser contexts pool size |
| `page_timeout` | `float` | `30.0` | Page load timeout |
| `request_timeout` | `float` | `10.0` | HTTP request timeout |
| `max_retries` | `int` | `3` | Retry attempts per page |
| `delay_between_requests` | `float` | `0.1` | Rate limiting delay |
| `scroll_pages` | `bool` | `False` | Scroll for lazy-loaded content |
| `extract_pdfs` | `bool` | `True` | Extract text from PDFs |
| `deduplicate_content` | `bool` | `True` | Skip duplicate pages |
| `use_browser_for_all` | `bool` | `False` | Always use Playwright |
| `headless` | `bool` | `True` | Run browser headless |

## Features

### Scraping Modes

- **Single**: Scrape one URL
- **Recursive**: Crawl entire site following internal links
- **Sitemap**: Parse sitemap.xml and scrape all URLs
- **URL List**: Scrape a provided list of URLs

### Content Extraction

- Automatic removal of scripts, styles, navigation, footers
- Clean text extraction with structure preservation
- Link extraction with markdown formatting
- PDF text extraction (optional dependency)

### Anti-Bot Features

- Realistic browser fingerprints
- Custom User-Agent and headers
- Stealth mode scripts
- Automatic retry with browser fallback

## Development

```bash
# Clone repository
git clone https://github.com/boxed-dev/trace.git
cd trace

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linter
ruff check .

# Type check
mypy trace_scraper
```

## License

MIT License - see [LICENSE](LICENSE) for details.
