Metadata-Version: 2.4
Name: crawldown
Version: 0.1.1
Summary: Crawl any website and save every page as organized Markdown files
Project-URL: Homepage, https://github.com/danilotpnta/crawldown
Project-URL: Issues, https://github.com/danilotpnta/crawldown/issues
Project-URL: Changelog, https://github.com/danilotpnta/crawldown/blob/main/CHANGELOG.md
Author-email: danilotpnta <falammal@gmail.com>
License: MIT License
        
        Copyright (c) 2026 danilotpnta
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: cli,crawling,documentation,markdown,web scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: crawl4ai>=0.8.6
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Description-Content-Type: text/markdown

# crawldown

[![CI](https://github.com/danilotpnta/crawldown/actions/workflows/ci.yml/badge.svg)](https://github.com/danilotpnta/crawldown/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/crawldown)](https://pypi.org/project/crawldown/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)

**Crawl any website and save every page as organized Markdown files.**

`crawldown` mirrors a website's URL structure into a local directory of `.md` files — perfect for archiving documentation, feeding content into RAG pipelines, or reading offline.

```
crawldown https://docs.example.com --output ./docs-mirror
```

```
docs-mirror/
├── index.md
├── getting-started/
│   ├── index.md
│   └── installation.md
└── api/
    ├── reference.md
    └── authentication.md
```

---

## Installation

```bash
pip install crawldown
```

Or with [uv](https://github.com/astral-sh/uv):

```bash
uv tool install crawldown
```

**First-time setup** — crawldown uses [crawl4ai](https://github.com/unclecode/crawl4ai) under the hood, which needs a browser installed once to handle JavaScript-rendered pages:

```bash
crawl4ai-setup
```

---

## Quickstart

### CLI

```bash
# Crawl an entire site
crawldown https://docs.example.com --output ./output

# Limit crawl depth
crawldown https://docs.example.com --output ./output --depth 2

# Add a delay between requests (seconds)
crawldown https://docs.example.com --output ./output --delay 0.5

# Only crawl URLs matching a pattern
crawldown https://docs.example.com --output ./output --include '/docs/*'

# Skip URLs matching a pattern
crawldown https://docs.example.com --output ./output --exclude '/api/*'

# Skip robots.txt enforcement
crawldown https://docs.example.com --output ./output --no-robots
```

### Python API

```python
import asyncio
from crawldown import crawl

asyncio.run(crawl("https://docs.example.com", output_dir="./output"))
```

With options:

```python
import asyncio
from crawldown import crawl
from crawldown.models import CrawlConfig

config = CrawlConfig(
    url="https://docs.example.com",
    output_dir="./output",
    max_depth=3,
    delay=0.5,
    respect_robots=True,
    include=["/docs/*"],   # only crawl these paths (glob)
    exclude=["/api/*"],    # skip these paths (glob)
)

asyncio.run(crawl(config))
```

---

## How it works

1. Starts at the given URL and fetches the page using [crawl4ai](https://github.com/unclecode/crawl4ai) (handles JavaScript-rendered pages).
2. Extracts all links that stay within the same domain and URL prefix.
3. Converts each page to Markdown and saves it at a path matching the URL structure.
4. Repeats for every discovered link up to `max_depth` (default: unlimited).

---

## Options

| Flag | Default | Description |
|------|---------|-------------|
| `--output`, `-o` | `./crawldown-output` | Directory to save Markdown files |
| `--depth`, `-d` | unlimited | Max link-follow depth |
| `--delay` | `0.0` | Seconds to wait between requests |
| `--include` | — | Only crawl paths matching this glob (repeatable) |
| `--exclude` | — | Skip paths matching this glob (repeatable) |
| `--no-robots` | off | Ignore `robots.txt` |
| `--version` | — | Show version and exit |

---

## Contributing

We welcome contributions of all kinds. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started.

---

## License

MIT — see [LICENSE](LICENSE).
