Metadata-Version: 2.4
Name: scrapy-ingest
Version: 1.1.1
Summary: Scrapy extension for database ingestion with job/spider tracking
Home-page: https://github.com/fawadss1/scrapy_item_ingest
Author: Fawad Ali
Author-email: fawadstar6@gmail.com
Project-URL: Documentation, https://scrapy-ingest.readthedocs.io/
Project-URL: Source, https://github.com/fawadss1/scrapy_item_ingest
Project-URL: Tracker, https://github.com/fawadss1/scrapy_item_ingest/issues
Keywords: scrapy,database,postgresql,mysql,web-scraping,data-pipeline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Framework :: Scrapy
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Database
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy>=2.18.0
Requires-Dist: psycopg2-binary>=2.9.12
Requires-Dist: itemadapter>=0.13.1
Requires-Dist: pytz>=2026.3
Requires-Dist: w3lib>=2.4.1
Requires-Dist: PyMySQL>=1.2.0
Provides-Extra: docs
Requires-Dist: sphinx>=5.0.0; extra == "docs"
Requires-Dist: sphinx_rtd_theme>=1.2.0; extra == "docs"
Requires-Dist: myst-parser>=0.18.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.19.0; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5.0; extra == "docs"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: flake8>=5.0.0; extra == "dev"
Requires-Dist: mypy>=0.991; extra == "dev"
Requires-Dist: pre-commit>=2.20.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-mock>=3.8.0; extra == "test"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Scrapy Ingest

A Scrapy addon that saves **items, requests, logs, and stats** to PostgreSQL or MySQL — with parent_url tracking, failed-request errors, and full job log capture (including `print()`).

## Install

```bash
pip install scrapy-ingest
```

## Minimal setup (settings.py)

Only the item pipeline is required — requests, logs, stats, parent_url, and error logging are enabled automatically:

```python
ITEM_PIPELINES = {
    "scrapy_ingest.pipelines.DbInsertPipeline": 300,
}

# PostgreSQL
DB_URL = "postgresql://user:password@localhost:5432/database"
# MySQL
# DB_URL = "mysql://user:password@localhost:3306/database"
# Or discrete fields (avoids URL encoding):
# DB_TYPE = "postgres"   # or "mysql"
# DB_HOST = "localhost"
# DB_PORT = 5432         # MySQL: 3306
# DB_USER = "user"
# DB_PASSWORD = "password"
# DB_NAME = "database"

# Optional
CREATE_TABLES = True     # auto-create tables on first run (default True)
JOB_ID = 1               # or omit; a unique id is generated per crawl
INGEST_BATCH_SIZE = 50   # flush when this many rows are buffered
```

Run your spider:

```bash
scrapy crawl your_spider
```

Log level follows Scrapy `LOG_LEVEL`.

## What is stored

| Table          | Contents                                                                                                |
|----------------|---------------------------------------------------------------------------------------------------------|
| `jobs`         | One row per crawl: `id`, unique `job_id` string, spider, status, start/finish, counts, items/min, stats |
| `job_items`    | JSON items (`crawled_at` added). `job_id` = `jobs.id` (CASCADE)                                         |
| `job_requests` | url, parent_url, parent_id, status, response_time, error, success. `job_id` = `jobs.id` (CASCADE)       |
| `job_logs`     | time, logger, level, message, exception. `job_id` = `jobs.id` (CASCADE)                                 |

Request `parent_url` is the page that scheduled the request (e.g. sitemap → product). Start URLs are `null`.

Data flushes on batch size, every 10s, and on engine/process stop.

When the spider closes, a crawl summary is printed (job, database, tables, items, requests, logs, errors, elapsed time) even if `LOG_LEVEL` is `ERROR`. Set `INGEST_SHOW_SUMMARY = False` to hide it.

## Troubleshooting

- Password has special characters like `@` or `$`?
  - In a URL, encode them: `@` -> `%40`, `$` -> `%24`.
  - Example: `postgresql://user:PAK%40swat1%24@localhost:5432/db`
  - Or use the discrete fields (no encoding needed).
- **Yield items** from callbacks (not only `return` inside a generator).

## Useful settings (optional)

- `DB_TYPE` (default: `postgres`) — `postgres` / `postgresql` or `mysql` / `mariadb` when building a URL from `DB_*` fields
- `INGEST_BATCH_SIZE` (default: `50`) — flush when this many items+requests+logs are buffered
- `INGEST_FLUSH_INTERVAL` (default: `10`) — periodic flush in seconds
- `INGEST_SHOW_SUMMARY` (default: `True`) — print crawl summary tables when the spider closes
- `CREATE_TABLES` (default: `True`) — create tables on startup
- `ITEMS_TABLE`, `REQUESTS_TABLE`, `LOGS_TABLE`, `JOBS_TABLE` — override table names
- `TIMEZONE` (default: `Asia/Karachi`) — timezone for `created_at`
- `JOB_ID` — omit to auto-generate a unique id (`Rs_Spider-178826754-a1b2c3`)

## Standalone components

If you only want part of the collection:

```python
# Items only
ITEM_PIPELINES = {"scrapy_ingest.pipelines.ItemsPipeline": 300}

# Requests only (parent_url + errors)
ITEM_PIPELINES = {"scrapy_ingest.pipelines.RequestsPipeline": 300}

# Logs only
EXTENSIONS = {"scrapy_ingest.extensions.LoggingExtension": 500}
```

## Links

- Docs: https://scrapy-ingest.readthedocs.io/
- Changelog: docs/development/changelog.rst
- Issues: https://github.com/fawadss1/scrapy_item_ingest/issues

## License

MIT License. See [LICENSE](LICENSE).
