Metadata-Version: 2.4
Name: scrapyd-logparser
Version: 0.1.0b1
Summary: A tool to parse Scrapyd logs for detailed statistics.
Project-URL: Homepage, https://github.com/jxlil/scrapyd-logparser
Author: Jalil SA
License: MIT
License-File: LICENSE
Keywords: logs,parser,scrapy,scrapyd,statistics,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: black>=24.1.1; extra == 'dev'
Requires-Dist: isort>=5.13.2; extra == 'dev'
Requires-Dist: pre-commit>=3.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# Scrapyd Log Parser

[![Python Version](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Status](https://img.shields.io/badge/Status-Beta-orange.svg)]()

A high-performance tool designed to parse **Scrapyd** logs and generate detailed statistics, providing deep insights that Scrapyd doesn't provide natively.

## Features

- **High Performance**: Leverages `ProcessPoolExecutor` for parallel log parsing, maximizing CPU utilization.
- **Organized Output**: Centralizes all parsed JSON data into a dedicated `scrapydlogparser` directory, mirroring your project's structure.
- **Loop Mode**: Automatically monitors and re-parses logs at configurable intervals.
- **Smart Cleanup**: Automatically removes orphaned JSON files when their corresponding log files are deleted.
- **Incremental Parsing**: Only processes new or modified logs by checking file size against existing data.
- **Error Detection**: Specifically detects critical unhandled errors and crashes, labeling them for easy identification.

## Quick Start

### Installation

Install in editable mode for development:

```bash
pip install -e .
```

### Basic Usage

Run the parser by pointing it to your Scrapyd logs directory:

```bash
scrapyd-logparser /path/to/scrapyd/logs
```

By default, this will:

1. Create a `scrapydlogparser/` directory inside your logs folder.
2. Generate individual `.json` files for every log, mirroring the project/spider structure.
3. Save a global `scrapydlogparser.json` summary in the same directory.

### CLI Options

| Option       | Shorthand | Description                                                | Default                                       |
| :----------- | :-------- | :--------------------------------------------------------- | :-------------------------------------------- |
| `--interval` | `-i`      | Interval in seconds for continuous monitoring (loop mode). | `5`                                           |
| `--output`   | `-o`      | Path to the summary JSON file.                             | `logs/scrapydlogparser/scrapydlogparser.json` |
| `--force`    | `-f`      | Forces a full re-parse of all log files.                   | `Disabled`                                    |
| `--json-dir` |           | Custom directory to store individual JSON files.           | `logs/scrapydlogparser/`                      |

## Advanced Usage

### Continuous Monitoring (Loop Mode)

To keep your statistics updated in real-time (e.g., every 60 seconds):

```bash
scrapyd-logparser ./logs --interval 60
```

## Data Structure

The tool transforms your standard Scrapyd logs into a clean, queryable JSON structure:

```text
logs/
├── project/
│   └── spider/
│       └── job.log
└── scrapydlogparser/
    ├── scrapydlogparser.json (Global Summary)
    └── project/
        └── spider/
            └── job.json (Detailed stats)
```
