Metadata-Version: 2.4
Name: scrapy_py
Version: 0.1.0
Summary: A simple web scraper library
Author-email: Arman Mondal <your@email.com>
Project-URL: Homepage, https://github.com/yourusername/my-scraper
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: requests
Requires-Dist: beautifulsoup4

# 🕷️ WebScraper

A Python library for **automating repetitive web scraping tasks** — built with batteries included.

## Features

| Module | What it does |
|---|---|
| `Scraper` | HTTP engine with retries, rate limiting, rotating User-Agents |
| `Parser` | HTML extraction — text, links, tables, images, meta tags |
| `Pipeline` | One-call scrape → parse → save for URL lists and paginated sites |
| `Scheduler` | Schedule recurring jobs (every N seconds/minutes/hours) |
| `Storage` | Save results as JSON or CSV, append, deduplicate |

---

## Installation

```bash
pip install requests beautifulsoup4 lxml
# Then copy the webscraper/ folder into your project
```

---

## Quick Start

### 1. Fetch a page

```python
from webscraper import Scraper, Parser

scraper = Scraper(delay=1.0)          # 1 second between requests
response = scraper.get("https://example.com")

parser = Parser(response.text)
print(parser.title())                  # → "Example Domain"
print(parser.links())                  # → ["https://..."]
```

### 2. Extract structured data

```python
data = parser.extract({
    "title": "h1",
    "price": ".product-price",
    "images": {"selector": "img", "attr": "src", "all": True},
})
# → {"title": "Product Name", "price": "$9.99", "images": [...]}
```

### 3. Scrape multiple pages automatically

```python
from webscraper import Pipeline, Storage

pipeline = Pipeline(
    scraper=Scraper(delay=1.5),
    storage=Storage("output"),
)

# Paginated scraping
results = pipeline.run_paginated(
    url_template="https://site.com/products?page={page}",
    start_page=1,
    end_page=10,
    extract_rules={
        "name":  "h2.product-title",
        "price": ".price",
        "link":  {"selector": "a.product-link", "attr": "href"},
    },
    output_file="products.json",
)
```

### 4. Schedule recurring scrapes

```python
from webscraper import Scheduler

scheduler = Scheduler()

@scheduler.every(minutes=15, max_runs=96)   # every 15 min for 24h
def monitor_prices():
    scraper = Scraper()
    resp = scraper.get("https://example.com/prices")
    parser = Parser(resp.text)
    # ... extract and save

scheduler.run()   # blocking
# or: scheduler.run(blocking=False)  # background thread
```

---

## API Reference

### `Scraper`

```python
Scraper(
    delay=1.0,           # seconds between requests
    delay_jitter=0.5,    # adds random 0–0.5s on top
    retries=3,           # auto-retry on 5xx / timeout
    timeout=30,          # request timeout
    headers={},          # custom headers
    proxies={},          # {"http": "...", "https": "..."}
    rotate_user_agents=True,
    verify_ssl=True,
)
```

| Method | Description |
|---|---|
| `.get(url)` | GET request |
| `.post(url, data, json)` | POST request |
| `.fetch_many(urls)` | Fetch list of URLs |
| `.crawl(start_url, max_pages)` | Follow links from start URL |

### `Parser`

```python
Parser(html_string)
```

| Method | Description |
|---|---|
| `.title()` | `<title>` text |
| `.text(selector, all=False)` | Text from CSS selector |
| `.attr(selector, attribute, all=False)` | Attribute value |
| `.links(base_url)` | All `<a href>` URLs |
| `.links_with_text()` | `[{"url": ..., "text": ...}]` |
| `.images(base_url)` | `[{"src": ..., "alt": ...}]` |
| `.table(index=0)` | Parse HTML table → list of dicts |
| `.all_tables()` | All tables |
| `.meta()` | All meta tags |
| `.og_data()` | Open Graph tags |
| `.body_text()` | All visible text |
| `.extract(rules)` | Multi-field extraction |
| `.extract()` rules | `{"field": "selector"}` or `{"field": {"selector": "...", "attr": "...", "all": bool}}` |

### `Pipeline`

```python
Pipeline(scraper=None, storage=None)
```

| Method | Description |
|---|---|
| `.run(urls, extract_rules, ...)` | Scrape a list of URLs |
| `.run_paginated(url_template, start, end, ...)` | Scrape paginated URLs |
| `.results` | Last run's results |

### `Scheduler`

```python
Scheduler()
```

| Method | Description |
|---|---|
| `.every(seconds, minutes, hours)` | Decorator to schedule a function |
| `.add_job(func, interval_seconds, ...)` | Register job programmatically |
| `.run(blocking=True)` | Start scheduler |
| `.stop()` | Stop scheduler |
| `.cancel(name)` | Cancel a job |
| `.status()` | List all jobs and their state |

### `Storage`

```python
Storage(output_dir="output")
```

| Method | Description |
|---|---|
| `.save_json(data, filename)` | Save as JSON |
| `.load_json(filename)` | Load JSON |
| `.append_json(new_data, filename)` | Append to JSON array |
| `.save_csv(data, filename)` | Save as CSV |
| `.load_csv(filename)` | Load CSV |
| `.append_csv(new_data, filename)` | Append to CSV |
| `.deduplicate(data, key)` | Remove duplicates by field |
| `.list_files()` | List output directory files |

---

## License

MIT
