Metadata-Version: 2.1
Name: scrapy-spiderstats-extension
Version: 0.0.2
Summary: Scrapy Spider Stats to MongoDB Extension
Home-page: https://github.com/crawlaio/scrapy-spiderstats-extension
Author: Sitoi
Author-email: 133397418@qq.com
License: MIT
Platform: UNKNOWN
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.6.0
Description-Content-Type: text/markdown
Requires-Dist: pymongo
Requires-Dist: twisted


# Scrapy SpiderStats Extension

将 Spider Stats 存储到 MongoDB 的扩展，可以用于爬虫监控和统计。

## 安装

```shell script
pip install scrapy-spiderstats-extension
```

## 使用

在 `settings.py` 配置文件中开启 `SpiderStats` 

```python
EXTENSIONS = {
    "scrapyspiderstats.SpiderStats": 0
}
STATS_MONGODB_URI = "mongodb://localhost:27017"
STATS_MONGODB_DB = "scrapy"
STATS_MONGODB_COL = "spiderstats"
```

## 存储结果

> 启动状态

```json
{
    "_id":"5fb23d9cbaf515d71d3a9c6c",
    "log_count/INFO":9,
    "start_time":"2020-11-16T08:51:40.705Z",
    "stats_id":"2b55df7b46a548269ca603bb7ad889b2",
    "spider_name":"test",
    "pages":0,
    "pagerate":0,
    "items":0,
    "itemrate":0,
    "record_time":"2020-11-16T08:51:40.706Z"
}
```


> 记录状态

```json
{
    "_id":"5fb23dd8baf515d71d3a9c6d",
    "log_count/INFO":12,
    "start_time":"2020-11-16T08:51:40.705Z",
    "stats_id":"2b55df7b46a548269ca603bb7ad889b2",
    "spider_name":"test",
    "pages":510,
    "pagerate":510,
    "items":0,
    "itemrate":0,
    "record_time":"2020-11-16T08:52:40.713Z",
    "log_count/DEBUG":1034,
    "scheduler/enqueued/redis":521,
    "scheduler/dequeued/redis":520,
    "downloader/request_count":520,
    "downloader/request_method_count/GET":520,
    "downloader/request_bytes":239235,
    "downloader/response_count":510,
    "downloader/response_status_count/200":510,
    "downloader/response_bytes":110675,
    "response_received_count":510,
    "downloader/exception_count":3,
    "downloader/exception_type_count/twisted-internet-error-TimeoutError":3,
    "retry/count":3,
    "retry/reason_count/twisted-internet-error-TimeoutError":3
}
```

> 完成状态

```json
{
    "_id":"5fb23e2ebaf515d71d3a9c6f",
    "log_count/INFO":16,
    "start_time":"2020-11-16T08:51:40.705Z",
    "stats_id":"2b55df7b46a548269ca603bb7ad889b2",
    "spider_name":"test",
    "pages":1000,
    "pagerate":6,
    "items":0,
    "itemrate":0,
    "record_time":"2020-11-16T08:54:06.125Z",
    "log_count/DEBUG":2015,
    "scheduler/enqueued/redis":1007,
    "scheduler/dequeued/redis":1007,
    "downloader/request_count":1007,
    "downloader/request_method_count/GET":1007,
    "downloader/request_bytes":463763,
    "downloader/response_count":1000,
    "downloader/response_status_count/200":1000,
    "downloader/response_bytes":216997,
    "response_received_count":1000,
    "downloader/exception_count":7,
    "downloader/exception_type_count/twisted.internet.error.TimeoutError":7,
    "retry/count":7,
    "retry/reason_count/twisted.internet.error.TimeoutError":7,
    "elapsed_time_seconds":145.420645,
    "finish_time":"2020-11-16T08:54:06.125Z",
    "finish_reason":"finished"
}
```

