Metadata-Version: 2.4
Name: crawlo
Version: 1.7.1
Summary: Crawlo: A high-performance asynchronous Python web crawling framework with distributed support.。
Home-page: https://github.com/crawl-coder/Crawlo.git
Author: crawl-coder
Author-email: crawlo@qq.com
License: BSD-3-Clause
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: aiohttp>=3.13.3
Requires-Dist: aiofiles>=25.1.0
Requires-Dist: aioredis>=2.0.1
Requires-Dist: chardet>=7.4.3
Requires-Dist: cssselect>=1.2.0
Requires-Dist: dateparser>=1.2.2
Requires-Dist: httpx[http2]>=0.27.0
Requires-Dist: curl-cffi>=0.13.0
Requires-Dist: lxml>=4.9.0
Requires-Dist: parsel>=1.9.1
Requires-Dist: pydantic>=2.11.7
Requires-Dist: python-dateutil>=2.9.0.post0
Requires-Dist: redis>=6.2.0
Requires-Dist: ujson>=5.9.0
Requires-Dist: w3lib>=2.3.1
Requires-Dist: rich>=13.0.0
Requires-Dist: astor>=0.8.0
Requires-Dist: watchdog>=3.0.0
Requires-Dist: psutil>=7.0.0
Requires-Dist: anyio>=4.3.0
Requires-Dist: httpcore>=1.0.5
Requires-Dist: msgpack>=1.0.0
Requires-Dist: pytz>=2024.2
Requires-Dist: yarl>=1.17.0
Provides-Extra: database
Requires-Dist: asyncmy>=0.2.11; extra == "database"
Requires-Dist: motor>=3.7.0; extra == "database"
Requires-Dist: pymongo>=4.11; extra == "database"
Requires-Dist: aiosqlite>=0.19.0; extra == "database"
Requires-Dist: asyncpg>=0.29.0; extra == "database"
Requires-Dist: clickhouse-connect>=0.7.0; extra == "database"
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "database"
Requires-Dist: happybase>=1.2.0; extra == "database"
Provides-Extra: sqlite
Requires-Dist: aiosqlite>=0.19.0; extra == "sqlite"
Provides-Extra: postgresql
Requires-Dist: asyncpg>=0.29.0; extra == "postgresql"
Provides-Extra: clickhouse
Requires-Dist: clickhouse-connect>=0.7.0; extra == "clickhouse"
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "elasticsearch"
Provides-Extra: hbase
Requires-Dist: happybase>=1.2.0; extra == "hbase"
Provides-Extra: bloom
Requires-Dist: pybloom-live>=0.3.1; extra == "bloom"
Provides-Extra: render
Requires-Dist: playwright<2,>=1.40; extra == "render"
Requires-Dist: camoufox<0.5,>=0.4.0; extra == "render"
Requires-Dist: DrissionPage>=4.1.0; extra == "render"
Provides-Extra: stealth
Requires-Dist: cloakbrowser[geoip]<0.5,>=0.4.0; extra == "stealth"
Provides-Extra: mcp
Requires-Dist: mcp>=1.0.0; extra == "mcp"
Requires-Dist: markdownify>=0.13.1; extra == "mcp"
Provides-Extra: monitoring
Requires-Dist: prometheus-client>=0.19.0; extra == "monitoring"
Provides-Extra: db-all
Requires-Dist: aiosqlite>=0.19.0; extra == "db-all"
Requires-Dist: asyncpg>=0.29.0; extra == "db-all"
Requires-Dist: clickhouse-connect>=0.7.0; extra == "db-all"
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "db-all"
Requires-Dist: happybase>=1.2.0; extra == "db-all"
Requires-Dist: asyncmy>=0.2.11; extra == "db-all"
Requires-Dist: motor>=3.7.0; extra == "db-all"
Requires-Dist: pymongo>=4.11; extra == "db-all"
Requires-Dist: aiosqlite>=0.19.0; extra == "db-all"
Requires-Dist: asyncpg>=0.29.0; extra == "db-all"
Requires-Dist: clickhouse-connect>=0.7.0; extra == "db-all"
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "db-all"
Requires-Dist: happybase>=1.2.0; extra == "db-all"
Provides-Extra: all
Requires-Dist: bitarray>=1.5.3; extra == "all"
Requires-Dist: PyExecJS>=1.5.1; extra == "all"
Requires-Dist: pymongo>=3.10.1; extra == "all"
Requires-Dist: redis-py-cluster>=2.1.0; extra == "all"
Requires-Dist: asyncmy>=0.2.11; extra == "all"
Requires-Dist: motor>=3.7.0; extra == "all"
Requires-Dist: pymongo>=4.11; extra == "all"
Requires-Dist: aiosqlite>=0.19.0; extra == "all"
Requires-Dist: asyncpg>=0.29.0; extra == "all"
Requires-Dist: clickhouse-connect>=0.7.0; extra == "all"
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "all"
Requires-Dist: happybase>=1.2.0; extra == "all"
Requires-Dist: playwright<2,>=1.40; extra == "all"
Requires-Dist: camoufox<0.5,>=0.4.0; extra == "all"
Requires-Dist: DrissionPage>=4.1.0; extra == "all"
Requires-Dist: cloakbrowser[geoip]<0.5,>=0.4.0; extra == "all"
Requires-Dist: mcp>=1.0.0; extra == "all"
Requires-Dist: markdownify>=0.13.1; extra == "all"
Requires-Dist: aiosqlite>=0.19.0; extra == "all"
Requires-Dist: asyncpg>=0.29.0; extra == "all"
Requires-Dist: clickhouse-connect>=0.7.0; extra == "all"
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "all"
Requires-Dist: happybase>=1.2.0; extra == "all"
Requires-Dist: asyncmy>=0.2.11; extra == "all"
Requires-Dist: motor>=3.7.0; extra == "all"
Requires-Dist: pymongo>=4.11; extra == "all"
Requires-Dist: aiosqlite>=0.19.0; extra == "all"
Requires-Dist: asyncpg>=0.29.0; extra == "all"
Requires-Dist: clickhouse-connect>=0.7.0; extra == "all"
Requires-Dist: elasticsearch<9.0.0,>=8.0.0; extra == "all"
Requires-Dist: happybase>=1.2.0; extra == "all"
Requires-Dist: prometheus-client>=0.19.0; extra == "all"

<p align="center">
  <img src="assets/logo.svg" alt="Crawlo Logo" width="150"/>
</p>

<h1 align="center">Crawlo</h1>

<p align="center">
  <strong>基于 asyncio 的现代化高性能 Python 异步爬虫框架</strong>
</p>

<p align="center">
  <strong>Python 3.8+</strong> · <strong>已适配 Python 3.14</strong>
</p>

<p align="center">
  <a href="README.md">中文</a> ·
  <a href="README.en.md">English</a>
</p>

<p align="center">
  <a href="#quick-start-zh">快速开始</a> ·
  <a href="#features-zh">核心特性</a> ·
  <a href="#docs-zh">文档</a> ·
  <a href="#examples-zh">示例</a>
</p>

---

## <a id="quick-start-zh"></a>✨ 快速开始（3步上手）

### 1. 安装
```bash
pip install crawlo
```

### 2. 创建爬虫
```bash
crawlo startproject myproject
cd myproject
crawlo genspider example example.com
```

### 3. 运行
```bash
crawlo run example
```

👉 **[查看5分钟快速上手教程 →](docs/getting-started/5min-quickstart.md)**

---

## <a id="features-zh"></a>🚀 核心特性

### ⚡ 高性能异步架构
- 基于 asyncio + aiohttp/httpx/curl-cffi 多种协议下载器
- 智能并发控制，连接池复用，自动优化吞吐量
- 支持 HTTP/2、TLS 指纹模拟（绕过 JA3 检测）

### 🛡️ 强大的反反爬能力
- **HybridDownloader**：5 级检测优先级，自动切换协议/浏览器引擎
- **Cloudflare 自动绕过**：检测挑战页面后自动切换隐身浏览器
- **4 种浏览器下载器**：Playwright / Camoufox / CloakBrowser / DrissionPage
- **BROWSER_* 统一配置层**：一套参数覆盖所有浏览器下载器
- **自适应选择器**：网站改版时自动重新定位元素（选择器自愈） — [使用指南 →](docs/guides/adaptive-selector.md)

### 🤖 AI 集成（MCP Server）
- Claude / Cursor 直接调用 Crawlo 抓取能力
- 三种抓取模式：`basic`（1-3s）→ `stealth`（3-10s）→ `max-stealth`（10s+）
- 浏览器单例池：stealth/max-stealth 模式复用实例
- 结构化错误返回：区分 `TIMEOUT` / `CONNECTION_ERROR` 等，含建议提示

### 📊 四级背压防线
- Engine 层请求生成控制 + QueueManager 策略驱动
- 智能增强：`IntelligentBackpressureCalculator` + `BackpressureMonitor`

### 📬 多渠道通知系统
- 5 种渠道：钉钉 / 飞书 / 企业微信 / 邮件 / 短信
- 30+ 预定义模板，异步发送，消息去重 + 窗口限制

### 🔄 三种部署模式

| 模式 | 配置 | 协调机制 | 适用场景 |
|------|------|---------|---------|
| **内存模式** | `RUN_MODE='standalone'` `QUEUE_TYPE='memory'` | 无（单机自动退出） | 开发调试、快速验证 |
| **多节点协作** ⭐ | `RUN_MODE='auto'` `QUEUE_TYPE='redis'` | 竞争消费（BZPOPMIN） | 多机并发，可接受任务丢失 |
| **分布式系统** | `RUN_MODE='distributed'` `QUEUE_TYPE='redis_stream'` | ACK + 心跳 + 故障转移 | 生产环境，任务可靠性高 |

> 三种模式的优先级模型完全一致，切换模式无需修改爬虫代码。
> [详细了解 →](docs/architecture-overview.md) · [生产部署指南 →](docs/deployment.md)

---

## <a id="docs-zh"></a>📚 文档

| 你是？ | 推荐阅读 |
|--------|---------|
| **新手** | [5分钟快速上手](docs/getting-started/5min-quickstart.md) → [安装指南](docs/getting-started/installation.md) |
| **开发者** | [配置指南](docs/guides/configuration/) → [调度指南](docs/guides/scheduling/) |
| **运维** | [三种部署模式详解](docs/architecture-overview.md) → [检查点系统](docs/concepts/checkpoint-guide.md) → **[生产部署指南](docs/deployment.md)** |

---

> ## 📖 [浏览完整文档](docs/index.md)
>
> 包含快速入门、使用指南、核心概念、API 参考、实战案例、FAQ 等全部内容。

---

## <a id="examples-zh"></a>💡 示例项目

查看 [`examples/`](examples/) 目录：
- **基础示例** - 快速上手
- **高级示例** - 复杂场景
- **生产级示例** - 可直接用于生产

---

## 🤝 贡献

欢迎提交 Issue 和 Pull Request！

---

## 📄 许可证

本项目采用 BSD 3-Clause 许可证 - 查看 [LICENSE](LICENSE) 文件了解详情。

---

<p align="center">
  <strong>⭐ 如果这个项目对你有帮助，请给我们一个 Star！</strong>
</p>
