Metadata-Version: 2.4
Name: metalist
Version: 0.7.2
Summary: A minimalist single-user note-taking app with SSR, fast in-memory tree operations, and efficient sync.
Project-URL: Homepage, https://github.com/evolvingstuff/metalist
Project-URL: Repository, https://github.com/evolvingstuff/metalist
Project-URL: Issues, https://github.com/evolvingstuff/metalist/issues
Requires-Python: <3.15,>=3.10
Description-Content-Type: text/markdown
Requires-Dist: argon2-cffi==25.1.0
Requires-Dist: cryptography==50.0.0
Requires-Dist: fastapi==0.141.1
Requires-Dist: httpcore==1.0.9
Requires-Dist: httpx==0.28.1
Requires-Dist: instructor==1.15.4
Requires-Dist: latex2mathml==3.81.0
Requires-Dist: loguru==0.7.3
Requires-Dist: Mako==1.3.12
Requires-Dist: nh3==0.3.6
Requires-Dist: openai==2.54.0
Requires-Dist: pillow==12.3.0
Requires-Dist: pydantic==2.13.4
Requires-Dist: python-dotenv==1.2.2
Requires-Dist: python-multipart==0.0.32
Requires-Dist: starlette==1.3.1
Requires-Dist: tqdm==4.70.0
Requires-Dist: tree-sitter==0.25.2
Requires-Dist: tree-sitter-javascript==0.25.0
Requires-Dist: typing_extensions==4.16.0
Requires-Dist: uvicorn==0.52.0
Requires-Dist: zxcvbn==4.5.0
Provides-Extra: dev
Requires-Dist: hypothesis==6.164.0; extra == "dev"
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"

# MetaList

A minimalist single-user note-taking app focused on server-side rendering (SSR), fast in-memory tree operations, and efficient sync/diff updates.

## Features
- Rich text editing (ContentEditable) with image support
- Drag-and-drop note reordering
- Real-time content saving
- Keyboard shortcuts / cheatsheet (press `?` in the app)
- Linked-list ordering model for efficient reorders
- Optional password protection + encryption at rest (AES-GCM)
- Multi-tab search contexts with server-persisted scroll/search state (survives browser restarts)
- Manual namespace backups/restores to a user-selected backup folder with retention controls

## Changes in 0.7.2

- Login, hydration, and workspace requests share one initialized browser-tab identity. Losing the `sessionStorage` copy while opening the workspace no longer causes the reported missing-tab-ID crash. The external trigger for that storage loss remains unconfirmed.
- AI chat preserves the edited note and includes its permitted tree, tags, and cached URL titles. Search redaction and cloud privacy restrictions apply to every node; blocked selections expose only their availability reason.
- Added on-demand AI help and menu actions, privacy previews, and Copy Response throughout assistant message bubbles. OpenAI is the supported inference provider.
- LLM regressions use current production prompts, default to five trials with cache-aware parallel scheduling, and can run only cases affected by prompt or skill changes.
- Improved Grammarly edit preservation and search suggestions that match the complete active clause.

## Changes in 0.6.3

- Restored HTTPS connection reuse and aligned the pending accept queue with the existing worker capacity, so a browser’s six-connection startup burst is admitted.
- Every page load or refresh checks PyPI asynchronously. A newly discovered release shows a dismissible notice linking to Version Info; typing `update` in the menu also finds it.
- Managed uv installations can update from Version Info, using the existing preflight, verified immutable backups, and namespace restart flow. Source checkouts and unmanaged installations explain why in-app installation is unavailable.
- Release validation now transfers every installed startup asset through concurrent persistent HTTP/HTTPS connections on all supported platforms and Python versions. Actual Edge startup over LAN-style HTTPS is also required on Windows before publication.

## Changes in 0.6.2

- Namespace startup allows two minutes, with progress every five seconds, instead of failing after 12 seconds on a busy machine. `METALIST_STARTUP_TIMEOUT_SECONDS` can increase the allowance. Failed children are stopped and reaped.
- Updates preserve the running Python interpreter and validate a disposable installation and namespace startup before stopping live servers. uv then installs the tested package/dependency versions from its cache without network access.
- Python 3.14 is supported. Command-line legacy imports also work when Python lacks Tk's native GUI component.
- Release validation now includes the real uv update, backups, and multi-namespace restart on Windows, macOS, and Linux across Python 3.10–3.14.

## Changes in 0.6.1

- Removed the unused performance overlay, including its menu option and background state updates, fixing a fatal redundant-state error during note rearranging. Obsolete overlay preferences and command usage are discarded automatically.

## Changes in 0.6.0

- Open notes and their children in multiple live, read-only floating windows. Drag and resize them, keep them visible across searches and tabs, and follow links or copy passwords. Saved changes refresh every half second.
- Double-click selects complete tags, including punctuation and prefixes, in search and tag bars.
- Choose sorting modes from the background right-click menu. Creation sorting uses the root note's creation time; last-updated sorting includes descendants and ignores reference display-form changes.
- Context menus have icons throughout. Full-screen notes have compact spacing, and both read-only views keep steady borders on hover.
- Fixed Version Info loading and state-transition errors during editing, focus changes, tab switching, and duplication.

## Changes in 0.5.0

- Sound support is removed entirely; reminders retain their visual behavior. The live database migrates to schema 9, with encrypted namespaces migrating after unlock. Existing backups remain unchanged. See [sound removal](docs/ui/sounds.md).
- Remove Formatting supports arbitrary spans within pasted headings. Whole-note Cmd+U removes leading/trailing blank lines and retains at most one empty line between sections.
- This release includes the security, recovery, resource-limit, state-ownership, and performance work documented in the [implementation and coverage map](docs/testing/coverage-map.md).

## Technology Stack

### Backend
- FastAPI
- SQLite (via stdlib `sqlite3`) with a guard-aware wrapper (`SafeSession`)
- Mako templates for SSR

### Frontend
- Vanilla JavaScript (no framework)
- HTML5 Drag and Drop API
- ContentEditable for rich text editing
- CSS custom properties for theming

### Testing
- Python/unit tests plus manual regression passes

## Architecture (High Level)
- Server renders the base page via Mako templates.
- The browser client drives interaction via `/api2` JSON endpoints.
- Passwordless notes load into memory at startup; encrypted notes load after login. Rendering/search and ordinary authentication use memory; a read guard restricts intentional SQLite access to explicit windows.

## Security Boundary

Password protection encrypts namespace data at rest. While a namespace is
unlocked, the server must hold its data-encryption key and decrypted working
data in process memory, and the browser holds the decrypted content currently
rendered in the page. A sufficiently privileged local process, debugger,
administrator, browser extension, or process-memory dump can therefore expose
an unlocked namespace.

Explicit logout removes the live key and purges decrypted runtime stores, but
it cannot guarantee forensic overwriting of memory previously allocated by the
Python or browser runtimes. Protect the host account and operating system, and
see the detailed [security architecture and threat model](docs/security/README.md).

## Development

### Setup
For a published one-off run with uv:
```bash
uvx metalist
```

For a persistent uv tool install:
```bash
uv tool install metalist
metalist
```

For a persistent uv installation, you can update from inside MetaList: type **update** in the menu, open **Version Info**, and choose **Update to …** when a newer release is available. Each browser load or refresh asks the server to check PyPI asynchronously. A reminder-style notice links to Version Info the first time a newer version is discovered; the announced version is remembered in that namespace's client preferences. A slow or failed check does not block the app. Version Info can retry a failed check.

The in-app updater checks the selected release, creates verified backups, and restarts every namespace. Keep Version Info open to follow progress and reload when ready; closing and reopening it in the same page resumes progress. Signing in again may be necessary. Source checkouts and unmanaged installations still check for releases, but show why in-app installation is unavailable. Update job records and logs live under `~/MetaList/update-jobs/` (or the configured data directory). If installation/restart fails after servers stop, inspect the updater output and run `metalist` on the server to restart it; there is no automatic rollback.

You can also update and restart MetaList with one cross-platform command:
```bash
metalist update
```
The updater checks the installed version against the latest PyPI release first. If MetaList is already current, it reports the installed version and leaves all running namespaces untouched. When an update is available, it checks installer prerequisites and uses uv to install the candidate into a disposable tool environment with the current base Python interpreter. It checks dependencies and starts a temporary namespace to verify HTTP readiness, version, and runtime assets. A failed preflight leaves the current installation and running namespaces unchanged. After preflight succeeds, it stops running namespaces, and creates and verifies a new backup of every namespace before installation can begin. Each archive includes the complete notes/settings database, the attachments database when present, and any legacy search-history database. Locked namespaces are backed up with their encrypted data and key metadata intact, without requiring a password.

Backups are saved to `~/MetaList/namespaces/<namespace>/backups/<namespace>-<timestamp>.metalist-backup.tar.gz`; the updater prints each verified path. Existing backups remain unchanged and are not pruned. If any backup fails, the update aborts with the current installation intact; run `metalist` to restart the stopped servers after resolving the failure.

Only after all backups pass does the updater hand off to an external PowerShell process on Windows or `/bin/sh` on macOS/Linux so the installed environment can unlock. It delegates installation to uv, pinning the current base interpreter and the exact tested package/dependency versions, using the cache populated during preflight with network access disabled. It then launches MetaList again, and reports the installed version (for example, `MetaList updated to v0.5.0.`). This protection requires an installed version containing the backup safeguard and applies to `metalist update` and the in-app update action; direct pip/uv install commands do not run it.

For pip, users can run `pip install metalist`. For a non-editable local install from this checkout, use `uv pip install .` or `pip install .` instead of the editable command below.

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --require-hashes -r requirements/ci.txt
python -m pip install --no-deps --no-build-isolation -e ".[dev]"

npm install
```

On Windows, create the same `.venv` with `python -m venv .venv` and activate it with `.venv\Scripts\Activate.ps1` in PowerShell. Run the same `python -m pip` commands after activation. Node dependencies are developer tooling; the application itself does not require Node.

### Run

Persistent data defaults to `~/MetaList`. Set `METALIST_DATA_DIRECTORY` to an absolute directory path before launch to use a separate data root, including namespaces, backups, certificates, logs, and runtime jobs. This is used by release tests to keep their disposable data separate from real notes.
The installed entrypoint starts or restarts every known namespace, prints their URLs, and exits:
```bash
metalist
```
For source-checkout compatibility, `python main.py` performs the same orchestration. Use `metalist work`, `python main.py --namespace work`, or `python main.py work` when you want one foreground namespace process.

Shell execution is disabled by default. To enable `@shell` for every namespace
started by the top-level orchestrator, use either:

```bash
metalist --enable-shell
python main.py --enable-shell
```

This flag is propagated to every namespace child process. The top-level process
prints a conspicuous shell-enabled banner before its namespace launch results.
Use `python main.py work --enable-shell` for one foreground namespace. Shell
routes remain restricted to loopback clients using a loopback request host,
even when the rest of MetaList is intentionally exposed to a LAN.

`metalist` and explicit single-namespace source runs bind to loopback at `127.0.0.1:8000` by default. This keeps a normal laptop launch off LAN and public interfaces unless remote access is explicitly configured.
On first startup, MetaList also auto-generates a self-signed TLS pair at `~/MetaList/certs/metalist-cert.pem` and `~/MetaList/certs/metalist-key.pem`, then enables HTTPS on the same bind host at port `8443`. If you already have real PEM files, point `METALIST_TLS_CERT` and `METALIST_TLS_KEY` at them instead. Set `METALIST_AUTO_GENERATE_TLS=0` only if you explicitly want HTTP-only startup.

Database selection:
- No explicit namespace on a single-namespace launch: `~/MetaList/namespaces/default/default.metalist.db`
- `--namespace work` or `METALIST_NAMESPACE=work`: `~/MetaList/namespaces/work/work.metalist.db`
- The related files DB is derived automatically, so `namespaces/work/work.metalist.db` uses `namespaces/work/work.metalist.files.db`
- Remembered launch ports are stored as plaintext metadata inside each namespace's main `*.metalist.db`
- Launch precedence is: explicit CLI flags > env vars > saved namespace profile; if a namespace has no saved profile, launch it once with explicit ports or configure ports from the UI
- Backups stay beside the namespace data under `~/MetaList/namespaces/work/backups/` and use one archive per snapshot with filenames like `work-<timestamp>.metalist-backup.tar.gz`
- The Backup Settings modal targets one user-selected backup folder and can include multiple namespaces in a single run
- Restoring `work` into `work` is the normal overwrite path; importing a backup under a different namespace name can create a new target namespace with automatically selected conflict-free ports.

Useful env flags:
- `CRASH_SERVER_ON_FAIL=1` (default): fail-fast on validation errors
- `API_PREFIX=/api2`: override API prefix (client assumes `/api2` by default)
- `METALIST_NAMESPACE=work`: select `~/MetaList/namespaces/work/work.metalist.db`
- `METALIST_HOST=127.0.0.1` (default): bind the main app to a specific interface; use a LAN IP or `0.0.0.0` only for intentional remote access
- `METALIST_ALLOWED_HOSTS=notes.example.com,192.168.1.20`: comma-separated public/LAN hostnames accepted in HTTP `Host` headers; loopback aliases and a specific non-wildcard `METALIST_HOST` are accepted automatically
- `METALIST_PORT=8000` (default): bind the main app to a different port
- `METALIST_HTTPS_PORT=8443`: override the HTTPS port when TLS is enabled
- `METALIST_TLS_CERT=/path/to/fullchain.pem` + `METALIST_TLS_KEY=/path/to/privkey.pem`: override TLS paths
- `METALIST_AUTO_GENERATE_TLS=0`: disable automatic creation of the default self-signed TLS pair
- default TLS paths: `~/MetaList/certs/metalist-cert.pem` and `~/MetaList/certs/metalist-key.pem`
- `METALIST_FORWARDED_ALLOW_IPS=127.0.0.1,::1` (default): trust proxy headers only from those reverse-proxy IPs
- `--enable-shell`: opt in to local `@shell` execution for this launch; shell routes require a loopback client and loopback request host, and the capability is never persisted in namespace data

### Remote Access / HTTPS
LAN or VPN access must be enabled explicitly. Prefer binding the machine's specific LAN address, which also adds that address to the accepted-host set:
```bash
METALIST_HOST=192.168.1.20 metalist
```
On a fresh machine, that first launch also creates the default TLS cert pair automatically. Then open either `http://192.168.1.20:8000` or `https://192.168.1.20:8443` from the other machine.

Starting with 0.7.1, an explicit `METALIST_HOST` or `METALIST_ALLOWED_HOSTS`
configuration is saved in `~/MetaList/network-settings.json` (under
`METALIST_DATA_DIRECTORY` when overridden). Subsequent source launches, plain
`metalist` launches, and updates reuse it. Environment values override and update
the corresponding saved settings. This file stores only the bind address and
allowed hosts; namespace ports remain in their launch profiles, and shell access
still requires `--enable-shell` on each launch. Invalid settings stop startup
with an error. Fresh installations continue to bind to localhost by default.
To switch back to local-only access, run `METALIST_HOST=127.0.0.1 metalist` once.
Older installed versions do not read this file; keep supplying their LAN settings
until upgrading to 0.7.1 or later.

To keep `@shell` available on the host laptop while allowing another laptop to
use normal MetaList features, bind both interfaces and explicitly allow the LAN
address:

```bash
METALIST_HOST=0.0.0.0 \
METALIST_ALLOWED_HOSTS=10.0.0.31 \
metalist --enable-shell
```

Use `http://127.0.0.1:<namespace-http-port>` on the host laptop when running
`@shell`. Other devices may use `https://10.0.0.31:<namespace-https-port>`, but
their shell start/status requests receive `403`.

Namespaced launch example:
```bash
metalist --namespace work --port 8001
```
This starts a separate process backed by `~/MetaList/namespaces/work/work.metalist.db` on `http://127.0.0.1:8001`.
Its backup snapshots live under `~/MetaList/namespaces/work/backups/` with filenames like `work-<timestamp>.metalist-backup.tar.gz`. New backups are versioned `.tar.gz` workspace archives; legacy `.bak` backups remain restorable.

After you launch a namespace once with explicit ports, MetaList remembers them in that namespace's main DB, so later you can use the shorthand:
```bash
metalist work
```
and MetaList will reuse the saved HTTP / HTTPS ports for `work`. The same applies to the default namespace: `metalist` will reuse the saved default-namespace profile.

Equivalent explicit launch, if you want it:
```bash
METALIST_HOST=0.0.0.0 \
METALIST_ALLOWED_HOSTS=192.168.1.20 \
METALIST_PORT=8000 \
METALIST_HTTPS_PORT=8443 \
metalist
```
From the other machine, open `https://<laptop-ip>:8443`.

If you already have a real certificate and key, use the same dual-listener flow:
```bash
METALIST_HOST=0.0.0.0 \
METALIST_ALLOWED_HOSTS=192.168.1.20 \
METALIST_PORT=8000 \
METALIST_HTTPS_PORT=8443 \
METALIST_TLS_CERT=/path/to/fullchain.pem \
METALIST_TLS_KEY=/path/to/privkey.pem \
metalist
```

If you want to rotate or regenerate the default self-signed pair manually, the helper script is still available:
```bash
generate-lan-cert.sh
```

When HTTPS is enabled:
- remote HTTP requests to `http://<laptop-ip>:8000` are redirected to HTTPS
- localhost HTTP requests still stay on plain `http://127.0.0.1:8000` so the laptop can keep using the non-TLS port

If TLS is terminated by a reverse proxy on the same machine instead, keep MetaList on loopback and let the proxy forward to it:
```bash
METALIST_HOST=127.0.0.1 \
METALIST_ALLOWED_HOSTS=notes.example.com \
METALIST_PORT=8000 \
METALIST_FORWARDED_ALLOW_IPS=127.0.0.1,::1 \
metalist
```

The reverse proxy must preserve the browser-facing `Host` header and set
`X-Forwarded-Proto`. MetaList never trusts `X-Forwarded-Host`; forwarded
scheme/client metadata is accepted only from `METALIST_FORWARDED_ALLOW_IPS`.
Do not widen that list beyond the actual proxy addresses.

### Legacy Import
`convert-from-legacy.py` replaces the SQLite database referenced by `app.config.DATABASE_URL`, clears its files sidecar, and imports notes from a legacy JSON export.

This is destructive. It deletes the existing notes DB and related files DB before rebuilding the namespace.

Example usage:
```bash
convert-from-legacy.py --input /path/to/legacy-export.json
```

Target a namespaced database during import:
```bash
convert-from-legacy.py --namespace work --input /path/to/legacy-export.json
```

If `--namespace`, `--port`, or `--https-port` are omitted, the import script prompts for them and saves the resulting launch profile inside the target namespace DB. That means a one-time import into `work` can immediately seed later shorthand launches like `metalist work`.

Namespace startup allows 120 seconds per child, reports progress every five seconds, and stops/reaps a failed child instead of leaving it running after reporting failure. Set `METALIST_STARTUP_TIMEOUT_SECONDS` to a positive number of seconds to allow more time on a heavily loaded machine. A crashed child still fails immediately. This is a startup allowance, not a guarantee that any machine will finish within two minutes.

The preflight is a disposable empty namespace; it cannot guarantee startup against every real database or prevent a later disk/process failure. If installation succeeds but real namespace startup fails, the new package remains installed: resolve the reported startup issue and run `metalist` again. No automatic code/database rollback is attempted, and backups remain unchanged.

### Publishing
For the real user-facing install flow:
```bash
uvx metalist
# or:
uv tool install metalist
metalist
```

This repo now packages itself under the PyPI distribution name `metalist`. Current releases support Python 3.10 through 3.14.

Recommended release path:
1. In the existing PyPI project `metalist`, configure GitHub Trusted Publishing for `evolvingstuff/metalist` and the workflow file `.github/workflows/publish-pypi.yml`.
2. Push the candidate commit to a branch and wait for the `Publish to PyPI` validation workflow to pass for that exact commit. GitHub-hosted Windows, macOS, and Linux runners install the built wheel and test application startup outside the source checkout on Python 3.10 through 3.14. Every matrix job also exercises the real `metalist update` path with uv, backups, interpreter preservation, and two-namespace restart against a local release index. The old-version fixture carries the candidate updater so this validates the mechanism being shipped; it does not retroactively fix older installed updaters. You do not need those operating systems locally. Archive verification also checks runtime files, including agent Markdown resources, against both the wheel and source distribution.
3. Only after every required check passes, create and push the release tag for that same commit. Any intervening change requires a new validation run before tagging. Branch and pull-request runs validate without publishing.
4. The tag workflow repeats validation and publishes the same artifacts tested by its full matrix. Manual publication also requires all checks to pass. Users can then run it with `uvx metalist`, install it persistently with `uv tool install metalist`, or install it with `pip install metalist`.

If `--input` is omitted, a file picker opens (when `tkinter` and its native `_tkinter` component are available).
Notes tagged with `@implies` are converted into ontology rules and are not imported as notes. Legacy rules that are invalid under the current ontology grammar are reported and skipped while valid rules continue importing.

### Run Tests

Python/unit test examples:
```bash
source .venv/bin/activate
.venv/bin/pytest
node --test tests/unit/*.mjs
.venv/bin/python -c "from pathlib import Path; import main; main._run_startup_sanity_gates(repo_root=Path.cwd())"
```

Source-checkout development launches can enable the mandatory Python and
JavaScript sanity gates with an ignored repository-root `.env` file:

```dotenv
METALIST_ENVIRONMENT=development
```

`main.py` prints the resolved environment at startup. An explicit process
environment value takes precedence over `.env`; only `development` and
`production` are accepted. When the variable and `.env` file are absent,
MetaList runs in production mode and skips the source sanity gates.

`TEST_MODE=1` and `POST /api2/test/reset` still exist for deterministic browser automation if we decide to add a new harness later, but Cypress is not part of the current workflow.

### Diagrams
Render Mermaid diagrams to PNGs:
```bash
npm run render-diagrams
```


## Maintenance and recovery

See the [dependency and release procedure](docs/security/supply-chain.md), [recovery runbook](docs/security/recovery.md), and [current test/implementation map](docs/testing/coverage-map.md). The dated [August code review](CODE_REVIEW.md) is historical.
