Metadata-Version: 2.4
Name: spidermoon
Version: 0.1.2
Summary: 面向 Python 爬虫的通用监控 SDK，支持请求监控、数据血缘、Kafka 输出、断网缓存和平台上报
Author: 刘叙
License-Expression: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: requests
Requires-Dist: requests>=2.28; extra == "requests"
Provides-Extra: aiohttp
Requires-Dist: aiohttp>=3.8; extra == "aiohttp"
Provides-Extra: scrapy
Requires-Dist: scrapy>=2.8; extra == "scrapy"
Provides-Extra: selenium
Requires-Dist: selenium>=4.0; extra == "selenium"
Provides-Extra: kafka
Requires-Dist: kafka-python>=2.0; extra == "kafka"
Provides-Extra: yaml
Requires-Dist: PyYAML>=6.0; extra == "yaml"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: requests>=2.28; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Provides-Extra: all
Requires-Dist: requests>=2.28; extra == "all"
Requires-Dist: aiohttp>=3.8; extra == "all"
Requires-Dist: scrapy>=2.8; extra == "all"
Requires-Dist: selenium>=4.0; extra == "all"
Requires-Dist: kafka-python>=2.0; extra == "all"
Requires-Dist: PyYAML>=6.0; extra == "all"
Dynamic: license-file

# Spider Monitor

`spidermoon` 是一个面向 Python 爬虫的通用监控 SDK。它可以安装到 Scrapy、Requests、aiohttp、Selenium 或自研爬虫项目中，统一采集爬虫运行指标、数据产物、文件下载、代码来源、数据质量、上报状态和爬虫运行维度。

后续爬虫监控平台的数据应全部来自该 SDK 的本地指标端点或主动上报协议。

## 核心能力

- 请求监控：请求数、成功率、失败率、响应时间、QPS。
- 本地指标：`/metrics`、`/health`。
- 主动上报：Reporter 定时 POST 到监控平台。
- 数据血缘：记录 item、文件、输出目标、来源 URL、代码位置。
- 数据质量：记录校验失败、重复、丢弃、解析失败。
- 上报诊断：Reporter 状态、端点测试、手动 flush。
- 爬虫运行维度：任务队列、任务重试、代理失败、封禁、限流、验证码、资源、日志、断点续爬。
- 框架接入：装饰器、上下文管理器、Requests、Scrapy、aiohttp、Selenium 适配器。

## 安装

基础安装：

```bash
pip install spidermoon
```

按技术栈安装：

```bash
pip install "spidermoon[requests]"
pip install "spidermoon[scrapy]"
pip install "spidermoon[aiohttp]"
pip install "spidermoon[selenium]"
pip install "spidermoon[kafka]"
pip install "spidermoon[yaml]"
pip install "spidermoon[all]"
```

> 当前版本已实现核心能力和 Requests/Scrapy/aiohttp/Selenium 接入；对应 extra 用于安装可选第三方依赖。

## 安装即用

常规生产爬虫只需要配置爬虫名、平台上报地址和维护人信息：

```bash
export MONITOR_NAME=product_spider
export MONITOR_REPORT_URL=https://monitor.company.com/api/v1/spidermoon/reports
export MONITOR_REPORT_TOKEN=your-platform-token
export MONITOR_OWNER=zhangsan
export MONITOR_OWNER_TEAM=crawler-team
export MONITOR_LABEL_ENV=prod
export MONITOR_LABEL_PROJECT=catalog
python spider.py
```

设置 `MONITOR_REPORT_URL` 后，SDK 会自动进入生产 profile：关闭本地 HTTP 服务、使用 SQLite 断网缓存、开启更大的批量上报、给上报周期增加 jitter、降低产物采样率、使用轻量代码来源采集，并关闭文件 hash。高吞吐爬虫仍然可以用环境变量或构造函数覆盖这些默认值。

生成完整生产模板：

```bash
spidermoon sample-config
```

对已有脚本做最小改动时，可以用命令行注入：

```bash
spidermoon run \
  --name product_spider \
  --report-url https://monitor.company.com/api/v1/spidermoon/reports \
  --owner zhangsan \
  --owner-team crawler-team \
  --label env=prod \
  -- python spider.py
```

## 最小示例

推荐一键接入：

```python
from spider_monitor import auto_monitor

monitor = auto_monitor(
    name="product_spider",
    report_endpoint="https://monitor.company.com/api/v1/spidermoon/reports",
    owner="zhangsan",
    owner_email="zhangsan@company.com",
    owner_team="crawler-team",
    run_type="daemon",
)
```

配置文件接入：

```json
{
  "name": "product_spider",
  "report_endpoint": "https://monitor.company.com/api/v1/spidermoon/reports",
  "owner": "zhangsan",
  "owner_email": "zhangsan@company.com",
  "owner_team": "crawler-team",
  "labels": {"env": "prod", "project": "catalog"},
  "resource_interval": 30
}
```

```bash
MONITOR_CONFIG_FILE=monitor.json python spider.py
```

资源和日志可直接进入同一条上报链路：

```python
import logging
from spider_monitor import SpiderMonitorLogHandler, auto_monitor

monitor = auto_monitor(name="product_spider")
monitor.start_resource_collection()

logger = logging.getLogger("product_spider")
logger.addHandler(SpiderMonitorLogHandler(monitor))
```

命令行注入：

```bash
spidermoon run --name product_spider -- python spider.py
```

需要手动控制时再使用完整 API：

```python
from spider_monitor import SpiderMonitor

monitor = SpiderMonitor(
    name="product_spider",
    labels={"env": "prod", "project": "catalog"},
    report_endpoint="https://monitor.company.com/api/v1/spidermoon/reports",
    owner="zhangsan",
    owner_email="zhangsan@company.com",
    owner_phone="13800000000",
    owner_team="crawler-team",
    run_type="manual",
    trigger_source="developer",
)

monitor.record_request(success=True, response_time=0.25)
monitor.record_item(
    item_type="product",
    item_id="100",
    source_url="https://example.com/products/100",
    fields={"id": "100", "name": "Keyboard", "price": 199.0},
)

print(monitor.get_stats())
monitor.flush()
monitor.finish_run(status="success", exit_code=0)
monitor.shutdown()
```

## 文档

- [开发与系统接入文档](docs/DEVELOPMENT.md)：SDK 架构、公共 API、上报协议、平台接入、测试、打包和验收标准。
- [使用教程](docs/USAGE.md)：安装、接入、上报平台、各类爬虫示例、生产建议和常见问题。
- [协议 JSON Schema](docs/protocol.schema.json)：平台接收 envelope 的基础结构。

## MVP 范围

首版建议稳定实现：

- `SpiderMonitor`
- `auto_monitor()`
- `record_request()`、`record_error()`、`record_page()`
- `record_item()`、`record_file()`
- `trace_context()`
- `get_stats()`、`get_reporter_status()`
- `flush()`、`reset()`、`shutdown()`
- `/metrics`、`/health`
- `monitor_request()`、`monitor_request_async()`、`monitor_context()`
- `MonitoredSession`
- `SpiderMonitorExtension`
- `monitored_client()`
- `MonitoredWebDriver`
- `MonitoredKafkaProducer`
- `spidermoon run`
- 基础 `operation` 事件：队列大小、任务重试、代理失败、封禁、限流、验证码、资源快照、日志、断点续爬
- JSON/YAML 配置文件：`config_file` 或 `MONITOR_CONFIG_FILE`
- 运行批次：`run_id`、`run_type`、`start_run()`、`finish_run()`
- 标准 Python 包打包：wheel、sdist、optional extras

## 平台接入

监控平台建议提供：

```text
POST /api/v1/spidermoon/reports
```

SDK 通过 `report_endpoint` 上报：

```python
monitor = SpiderMonitor(
    name="product_spider",
    report_endpoint="https://monitor.company.com/api/v1/spidermoon/reports",
)
```

协议详见 [开发与系统接入文档](docs/DEVELOPMENT.md) 的“上报协议”章节。

多服务器、多爬虫会自动按 `client_id = <spider_name>@<hostname>:<pid>` 区分实例，平台按 `name`、`client_id`、`owner` 和 labels 聚合展示。

建议生产环境必须配置维护人信息，便于告警时直接定位到人：

```bash
export MONITOR_OWNER=zhangsan
export MONITOR_OWNER_EMAIL=zhangsan@company.com
export MONITOR_OWNER_PHONE=13800000000
export MONITOR_OWNER_TEAM=crawler-team
```
