Metadata-Version: 2.4
Name: loclore
Version: 0.1.0
Summary: LoCloRe (local clone rebrand): clone a website into a self-contained local folder, with optional text rebranding
Author: stefan.insam
Author-email: stefan.insam <stefan.insam@netgo.de>
License-Expression: GPL-3.0-only
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP :: Site Management
Classifier: Topic :: Utilities
Requires-Dist: beautifulsoup4>=4.15.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: lxml>=6.1.3
Requires-Dist: rich>=15.0.0
Requires-Dist: typer>=0.27.2
Requires-Python: >=3.14
Project-URL: Repository, https://github.com/ramsesoriginal/LoCloRe
Project-URL: Issues, https://github.com/ramsesoriginal/LoCloRe/issues
Description-Content-Type: text/markdown

<div align="center">
  <img src="https://raw.githubusercontent.com/ramsesoriginal/LoCloRe/main/images/logo.svg" alt="LoCloRe" width="200">

  <p><strong>Clone a page into a self-contained local folder, and rebrand it on the way in.</strong></p>

  [![PyPI](https://img.shields.io/pypi/v/loclore?color=blue)](https://pypi.org/project/loclore/)
  [![CI](https://github.com/ramsesoriginal/LoCloRe/actions/workflows/ci.yml/badge.svg)](https://github.com/ramsesoriginal/LoCloRe/actions/workflows/ci.yml)
  [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE)
  [![Python 3.14+](https://img.shields.io/badge/python-3.14%2B-3776AB?logo=python&logoColor=white)](pyproject.toml)
  [![uv](https://img.shields.io/badge/uv-managed-DE5FE9?logo=uv&logoColor=white)](https://github.com/astral-sh/uv)
  [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
  [![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
  [![GitHub stars](https://img.shields.io/github/stars/ramsesoriginal/LoCloRe?style=social)](https://github.com/ramsesoriginal/LoCloRe)
</div>

---

**LoCloRe** (*lo*cal *clo*ne *re*brand) downloads a page(HTML, CSS, JS,
images, fonts) into one folder, rewrites every same-origin reference to a
relative path, and serves it locally so it looks and behaves like the real
site with no network access required (aside from any third-party CDN
scripts, which are deliberately left alone).

On the way in, it can apply find/replace "rebrand" rules to every page and
stylesheet: handy for previewing a rename, a version bump, or a new logo
against a real site before touching production.

## Installation

```bash
uv tool install loclore
# or: pipx install loclore
# or: pip install loclore
```

Unreleased changes straight from `main`:

```bash
uv tool install git+https://github.com/ramsesoriginal/LoCloRe
```

Remove it later with `uv tool uninstall loclore`.

## Usage

```bash
loclore https://example.com
```

Downloads the homepage and everything it references into
`./clone-example.com/`, then serves that folder at a local
`http://127.0.0.1:<port>/` URL and opens it in your browser (see
[Viewing the result](#viewing-the-result)).

Rebrand rules and a custom output folder:

```bash
loclore https://example.com \
  -o ./preview \
  -r "Version 0.1: beta Version=Version 0.2: release candidate" \
  -r "Company XYZ=Company XYZ: ABC" \
  -r "logo.png=logo-new.png"
```

Follow internal links one hop deep, to clone a small site instead of one page:

```bash
loclore https://example.com --depth 1 --max-pages 40
```

### Options

| Flag | Description |
| --- | --- |
| `URL` (positional) | page to clone, e.g. `example.com` or `https://example.com` |
| `-o`, `--output PATH` | output directory (default: `./clone-<hostname>`) |
| `-c`, `--config PATH` | TOML config file; see [Config file](#config-file) |
| `-d`, `--depth N` | follow internal links this many hops beyond the start page (default: `0`, homepage only) |
| `--max-pages N` | safety cap on pages crawled, regardless of depth (default: `50`) |
| `--concurrency N` | max concurrent HTTP requests (default: `8`) |
| `--timeout SECONDS` | per-request timeout (default: `15`) |
| `--delay SECONDS` | minimum seconds between request starts; raise if a site rate-limits you (default: `0`) |
| `--retries N` | retries for 429/502/503/504, honoring `Retry-After`; see [If a site blocks or rate-limits you](#if-a-site-blocks-or-rate-limits-you) (default: `2`) |
| `--include-domain HOST` | extra hostname to treat as first-party, e.g. a CDN subdomain you control (repeatable) |
| `-r`, `--rebrand FIND=REPLACE` | text mapping to apply everywhere; see [How rebranding works](#how-rebranding-works) (repeatable) |
| `--rebrand-file PATH` | JSON/TOML file of rebrand mappings, `{find: replace}` or a list of `{find, replace, regex?}` |
| `--user-agent STRING` | custom `User-Agent` header |
| `-H`, `--header "Name: Value"` | extra HTTP header to send with every request (repeatable) |
| `--overwrite` / `--no-overwrite` | allow writing into a non-empty output directory (default: no; nothing is ever deleted either way) |
| `--serve` / `--no-serve` | serve the output over local HTTP and open a browser when done (default: yes) |
| `--port N` | port for the local server (default: automatically chosen, so it never fails on a conflict) |
| `--open-browser` / `--no-open-browser` | automatically open a browser tab when serving (default: yes) |
| `-v`, `--verbose` / `--no-verbose` | print extra diagnostic detail (default: no) |
| `--version` | show the version and exit |

## Config file

For more than a couple of rules, use a TOML file instead, see
[loclore.example.toml](loclore.example.toml) for every field:

```bash
cp loclore.example.toml myproject.toml
# edit myproject.toml
loclore --config myproject.toml
```

CLI flags override the config file: keep rules in the file and still do
`loclore --config myproject.toml --depth 1` for a one-off deeper crawl.

Rebrand rules can also live in their own JSON/TOML file (share a rule set
across configs, or generate one from another tool), see
[rebrand.example.json](rebrand.example.json):

```bash
loclore https://example.com --rebrand-file rebrand.example.json
```

## How rebranding works

Every rule is a plain substring match by default (`find` -> `replace`),
applied to every downloaded HTML/CSS/JS/SVG file, in a single pass over the
*original* text:

- One rule's output never becomes another rule's input, no cascading
  replacements.
- If two rules could match at the same spot (`"Company XYZ"` and
  `"Company XYZ: ABC"`), the longer `find` always wins, regardless of
  declaration order.

Set `regex = true` (TOML) / `"regex": true` (JSON) to match a pattern
instead of literal text. The replacement is still inserted as plain text
(no `\1`-style backreferences).

Renaming a referenced file (`logo.png` -> `logo-new.png`) only rewrites the
*reference*. It doesn't fetch or invent `logo-new.png`. Drop your
replacement asset into the output folder at that path yourself.

Every run ends with a table of each rule and its match count, so a typo'd
`find` (0 matches) is obvious immediately.

## What gets downloaded, what doesn't

Same-origin: images, stylesheets (incl. `@import`/`url(...)`-referenced
fonts/images), scripts, inline `<style>`/`style=""` backgrounds, `srcset`
variants, favicons/manifest icons, all downloaded and rewritten to
relative paths.

Cross-origin (CDN libraries, embedded widgets, tracking scripts,
`<iframe>`s): left as the original absolute URL, untouched. That's the
whole CDN rule: same-origin gets cloned, cross-origin stays pointed at the
live internet. Own static assets on a separate host? Add it with
`--include-domain static.example.com` (repeatable, or `include_domains` in
the config file).

Links beyond `--depth`/`--max-pages`, or that failed to download, are left
as absolute URLs, the clone degrades to the live site instead of a dead
link.

## Output layout

Everything lives under the output directory, mirroring each asset's URL
path (`/static/css/app.css` -> `<output>/static/css/app.css`); pages become
`index.html` / `<path>.html`. LoCloRe never touches anything outside that
one directory. `--include-domain` assets are namespaced under
`<output>/_external/<hostname>/...`.

Cloned a sub-page, not the site root? Its content naturally lands at e.g.
`about.html`, not `index.html`. LoCloRe adds a tiny redirect `index.html`
at the root pointing to it (only when nothing else already occupies that
path), so opening or serving the folder always lands on the clone.

A non-empty existing output directory is refused unless `--overwrite` is
passed, which never deletes anything, it just allows writing into it.

## Viewing the result

`file://` blocks or restricts things pages rely on: `fetch`/XHR, ES
modules, some fonts. So by default, once a clone finishes, LoCloRe serves
the output folder locally (Python's standard-library `http.server`, the
same thing `python -m http.server` runs) and opens it in your browser.
Runs until you stop it with Ctrl+C.

```bash
loclore https://example.com --port 8080          # fixed, predictable port
loclore https://example.com --no-open-browser    # don't launch a browser tab
loclore https://example.com --no-serve           # just clone, exit immediately (scripts/CI)
```

## If a site blocks or rate-limits you

Requests already carry a realistic browser `User-Agent` plus
`Accept`/`Accept-Language` headers by default. Rate-limit-ish responses
(429/502/503/504) are retried automatically with backoff, honoring
`Retry-After` when sent. If a site still pushes back:

```bash
# Space out requests, give retries more room.
loclore https://example.com --delay 0.5 --retries 4

# A specific User-Agent and/or extra headers (e.g. a session cookie).
loclore https://example.com \
  --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
  -H "Accept-Language: de-DE,de;q=0.9" \
  -H "Cookie: session=..."
```

A `403`/persistent block (vs. a `429`) usually means the site decided
*you*, not just your request rate, are unwelcome. Retrying won't help;
`--delay` and closer-to-a-real-browser headers are what to adjust. Also:
repeatedly cloning the same page in quick succession can itself trip a
rate limiter, independent of anything above.

## Known limitations

- Static clone only: JavaScript is copied as-is but not executed, so
  content rendered client-side after page load won't appear.
- Lazy-loaded images that only populate a `data-src`-style attribute (no
  plain `src`) aren't picked up.
- References inside downloaded `.js` files aren't rewritten (rebrand
  text-replacement still applies); HTML/CSS/SVG references are fully
  rewritten.

## Development

```bash
uv sync
uv run loclore https://example.com --no-serve  # smoke test

uv run pytest             # tests
uv run ruff check .       # lint
uv run ruff format .      # format
uv run mypy src tests     # type check
```

Every push and pull request runs this same lint/type-check/test suite via
[GitHub Actions](.github/workflows/ci.yml), across Linux and Windows.

## Changelog

See [CHANGELOG.md](CHANGELOG.md).

## License

[GPL-3.0](LICENSE): see [LICENSE](LICENSE) for the full text.
