Metadata-Version: 2.5
Name: mineworker
Version: 0.23.0
Summary: 一个上手简单、结构清晰的 Python 爬虫框架（对标 feapder）
Project-URL: Homepage, https://github.com/apersonw/mineworker
Project-URL: Repository, https://github.com/apersonw/mineworker
Project-URL: Documentation, https://apersonw.github.io/mineworker/
Project-URL: Changelog, https://github.com/apersonw/mineworker/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/apersonw/mineworker/issues
Author-email: Lucas <schummshelly26@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: crawler,feapder,scraping,spider,爬虫
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: bitarray>=2.9
Requires-Dist: httpx>=0.27
Requires-Dist: loguru>=0.7
Requires-Dist: parsel>=1.9
Requires-Dist: w3lib>=2.1
Provides-Extra: all
Requires-Dist: curl-cffi>=0.16; extra == 'all'
Requires-Dist: dbutils>=3.0; extra == 'all'
Requires-Dist: elasticsearch>=8.0; extra == 'all'
Requires-Dist: jinja2>=3.1; extra == 'all'
Requires-Dist: kafka-python>=2.0; extra == 'all'
Requires-Dist: playwright>=1.44; extra == 'all'
Requires-Dist: prometheus-client>=0.20; extra == 'all'
Requires-Dist: psycopg[binary,pool]>=3.1; extra == 'all'
Requires-Dist: pymongo>=4.7; extra == 'all'
Requires-Dist: pymysql>=1.1; extra == 'all'
Requires-Dist: redis>=5.0; extra == 'all'
Requires-Dist: typer>=0.12; extra == 'all'
Provides-Extra: cli
Requires-Dist: jinja2>=3.1; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: curl
Requires-Dist: curl-cffi>=0.16; extra == 'curl'
Provides-Extra: dev
Requires-Dist: curl-cffi>=0.16; extra == 'dev'
Requires-Dist: dbutils>=3.0; extra == 'dev'
Requires-Dist: elasticsearch>=8.0; extra == 'dev'
Requires-Dist: fakeredis[lua]>=2.20; extra == 'dev'
Requires-Dist: jinja2>=3.1; extra == 'dev'
Requires-Dist: kafka-python>=2.0; extra == 'dev'
Requires-Dist: mongomock>=4.1; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: playwright>=1.44; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: prometheus-client>=0.20; extra == 'dev'
Requires-Dist: psycopg[binary,pool]>=3.1; extra == 'dev'
Requires-Dist: pymongo>=4.7; extra == 'dev'
Requires-Dist: pymysql>=1.1; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest-httpserver>=1.0; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: redis>=5.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: typer>=0.12; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.25; extra == 'docs'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch>=8.0; extra == 'elasticsearch'
Provides-Extra: kafka
Requires-Dist: kafka-python>=2.0; extra == 'kafka'
Provides-Extra: metrics
Requires-Dist: prometheus-client>=0.20; extra == 'metrics'
Provides-Extra: mongo
Requires-Dist: pymongo>=4.7; extra == 'mongo'
Provides-Extra: mysql
Requires-Dist: dbutils>=3.0; extra == 'mysql'
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: psycopg[binary,pool]>=3.1; extra == 'postgres'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Provides-Extra: render
Requires-Dist: playwright>=1.44; extra == 'render'
Description-Content-Type: text/markdown

# MineWorker

[![PyPI](https://img.shields.io/pypi/v/mineworker)](https://pypi.org/project/mineworker/)
[![Python](https://img.shields.io/pypi/pyversions/mineworker)](https://pypi.org/project/mineworker/)
[![CI](https://github.com/apersonw/mineworker/actions/workflows/ci.yml/badge.svg)](https://github.com/apersonw/mineworker/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![Docs](https://img.shields.io/badge/docs-apersonw.github.io-teal)](https://apersonw.github.io/mineworker/)

一个上手简单、结构清晰的 Python 爬虫框架，对标 [feapder](https://github.com/Boris-code/feapder)：
你只写 `start_requests` 和 `parse`，框架负责调度、下载、重试、去重、批量落库。

> **0.15.0** —— 单机（`AirSpider`）到分布式（`Spider` / `TaskSpider` / `BatchSpider`）全部可用，
> 并支持[浏览器 TLS 指纹伪装](https://apersonw.github.io/mineworker/anti-bot/)。
> 变更见 [CHANGELOG](CHANGELOG.md)，后续规划见 [Roadmap](https://apersonw.github.io/mineworker/roadmap/)。

## 安装

```bash
pip install mineworker            # 核心
pip install "mineworker[all]"     # 含渲染 / 各类存储 / Redis / CLI / 指标 / 指纹伪装
```

也可以按需装单项：`render` · `mongo` · `mysql` · `postgres` · `elasticsearch` · `kafka` ·
`redis` · `cli` · `metrics` · `curl`。

## 快速开始

```bash
pip install "mineworker[cli]"
mineworker create -p news_crawler
cd news_crawler && python main.py
```

或直接写：

```python
import mineworker as mw


class NewsSpider(mw.AirSpider):
    def start_requests(self):
        yield mw.Request("https://news.ycombinator.com/", callback=self.parse)

    def parse(self, request, response):
        for a in response.css("span.titleline > a"):
            yield {"title": a.css("::text").get(), "url": a.css("::attr(href)").get()}


NewsSpider().start()
```

## 示例

[`examples/`](examples/) 里有可以直接跑的完整例子：

```bash
python examples/books_toscrape.py   # 两级抓取：列表页翻页 → 详情页
```

## 文档

完整文档：**<https://apersonw.github.io/mineworker/>**
（本地预览：`pip install "mineworker[docs]" && mkdocs serve`）

设计与实施计划见 [IMPLEMENTATION_PLAN.md](IMPLEMENTATION_PLAN.md)。

## 能力一览

| | |
|---|---|
| 运行时 | 单进程多线程、优先级队列、结束检测、`Ctrl-C` 优雅排空、崩溃 dump + `retry` 回放 |
| 网络 | `Request`/`Response`（httpx + parsel）、自动重试、`validate`/`failed_request` 钩子、随机 UA、会话复用 |
| 礼貌性 | `robots.txt`（含 `Crawl-delay`）、per-domain 限速 + 单域并发上限、**跨节点全局限速**（Redis）、429/503 读 `Retry-After` 整域降速 |
| 稳健性 | 状态码策略（错误页不当数据入库）、指数退避 + 抖动、同域连续失败熔断、运行时长上限、响应体大小上限 + Content-Type 白名单、`SIGINT`/`SIGTERM` 优雅停止、**任务租约 + 落库后才销账**（节点被硬杀，任务能回收、数据不丢） |
| 分布式 | `Spider`：Redis 队列 + 布隆去重 + 断点续爬 + 种子一次性锁 + 多节点心跳结束检测 |
| 任务驱动 | `TaskSpider` 从任务源持续消费；`BatchSpider` 周期批次采集（任务表状态机 + 进度追踪 + 防丢） |
| 数据 | `Item`/`UpdateItem`、`Pipeline`（Console/CSV/MongoDB/MySQL/PostgreSQL/Elasticsearch/Kafka）、请求级 + Item 级去重（布隆/精确） |
| 渲染 | `Request(render=True)` —— Playwright 渲染池、`wait_for`/`render_time`/`render_script` |
| 反爬 | TLS / HTTP2 指纹伪装（`impersonate` 真实浏览器）、Cloudflare / Akamai 挑战页识别 |
| 扩展 | 下载中间件链、**代理池 / 账号池**（耗尽时不会悄悄降级成直连或匿名）、掉登录自动换号 |
| 观测 | Prometheus exporter、卡死/失败率告警（**飞书 / 钉钉 / 企业微信 / 邮件**，发送失败会记录）、`debug=True` |
| 开发体验 | **响应缓存**（重跑读本地文件，不再反复打目标站） |
| 工具 | `mineworker create/shell/retry/cache`，`create -i --table` 读表结构反射生成 Item |

## 开发

```bash
conda env create -f environment.yml && conda activate mineworker
pre-commit install
pytest && ruff check . && mypy
```

CI（GitHub Actions）：ruff + mypy(strict) + pytest（Python 3.10–3.13）+ mkdocs build。

## License

[MIT](LICENSE)
