Metadata-Version: 2.4
Name: logryn
Version: 1.0.0b1
Summary: High-availability, async-safe logging framework for Python pipelines using MariaDB.
Author-email: Elad Segev <200956837+elad-segev@users.noreply.github.com>
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: sqlalchemy
Requires-Dist: pandas

# Logryn

**High-availability, async-safe logging for Python pipelines that aren't allowed to go down.**

[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-orange.svg)](#contributing)

---

## Why Logryn?

Standard logging setups are built for scripts that run for minutes, not services that run for months.

Point a naive logger at a database from a multi-threaded pipeline and you'll eventually hit one of these:

- **A blocked event loop.** Writing to a DB from inside an async app without care means your `await` chain stalls on I/O that has nothing to do with your actual work.
- **A flood of connections.** Every worker thread opening its own DB connection is a fast way to exhaust `wait_timeout` and get your pipeline throttled or rejected by the DB server.
- **Silent memory leaks.** If the database blips and logs pile up faster than they drain, an unbounded queue turns into an OOM kill a few weeks into production.
- **Data loss on crash.** `SIGTERM` arrives, the process dies, and whatever was sitting in the log buffer disappears — right when you need it most.

Logryn exists to make these non-issues. It runs a single async writer loop behind a thread-safe bridge, caps its own memory footprint under load, and flushes everything it's holding before the process actually exits.

---

## Key Features

* **Thread-Boundary Safety** : Worker threads do not access the event loop directly. Each `write()` call is delegated through `loop.call_soon_threadsafe`, preventing concurrency conflicts between synchronous producers and the asynchronous writer.
* **Smart Backpressure and Batching** : Logs are stored in a bounded `asyncio.Queue` with a default capacity of 20,000 entries and flushed to the database in batches. When log production exceeds database throughput, the oldest entries are discarded to maintain a bounded memory footprint and prevent unbounded queue growth.
* **Graceful Shutdown** : `SIGINT` and `SIGTERM` are handled at the event-loop level, with a signal-based fallback on Windows. During shutdown, Logryn records a final `"System Interruption"` event, drains the queue, and flushes the remaining entries to the database before the process exits.
* **Efficient DB Pooling** : Database engines are cached using `lru_cache`, allowing repeated calls to reuse existing connection pools rather than creating new ones. Pools use `pool_pre_ping` and `pool_recycle` to reduce failures caused by stale connections and long idle periods, including `wait_timeout` errors.
* **Security by Default** : Database queries are executed through SQLAlchemy's parameterized `text()` interface. Dynamic filters and `WHERE` clauses are constructed using an operator allow-list containing `=`, `!=`, `>`, `<`, `>=`, `<=`, `LIKE`, and `IN`, preventing unsupported operators from being incorporated into queries.
* **Global Crash Visibility** : Unhandled exceptions in both the main thread and worker threads are captured through `sys.excepthook` and `threading.excepthook` and logged automatically, ensuring that unexpected failures are not silently ignored.

---

## Architecture Overview

Logryn follows a **producer–consumer** pattern purpose-built to bridge sync and async worlds safely:

```
┌──────────────┐      ┌────────────────┐      ┌─────────────┐      ┌──────────┐
│ Worker Thread│      │  call_soon_    │      │  Async      │      │ MariaDB  │
│  (sync code) │  ->  | threadsafe     │  ->  │  Queue      │  ->  │  Writer  │  ->  Database
│  logger.write│      │  (thread-safe  │      │ (bounded,   │      │  Loop    │
└──────────────┘      │   bridge)      │      │  drop-oldest│      │ (batched)│
                      └────────────────┘      └─────────────┘      └──────────┘
```

Any number of synchronous threads can call `logger.write(...)` freely. Each call is marshalled onto the event loop safely, buffered, and flushed to the database in batches by a single background worker — keeping the DB connection count and load predictable regardless of how many threads are producing logs.

---

## Installation

```bash
pip install logryn
```

Or, for now, install directly from source:

```bash
git clone https://github.com/elad-segev/logryn.git
cd logryn
pip install -e .
```

> Logryn requires `sqlalchemy`, `mariadb` connector support (`mariadb+mariadbconnector`), and `pandas` (for `ThreadConfigManager`).

---

## Quick Start

```python
import asyncio
import threading
from logryn import MainLog, setup_global_error_handler

# 1. Configure your database connection
db_config = {
    "user": "logryn_user",
    "password": "super-secret",
    "host": "127.0.0.1",
    "port": 3306,
}

# 2. Create the logger — this owns the log table + async writer
logger = MainLog(
    log_name="system_logs",
    database_name="my_pipeline_db",
    db_config=db_config,
    timeout=5.0,   # how long the worker waits for new logs before checking batch
    batch=50,      # flush to DB once this many logs are buffered
)


async def main():
    # 3. Boot the DB table + background writer loop
    ok, msg = logger.initialize_db()
    if not ok:
        raise RuntimeError(msg)

    # 4. Wire up SIGINT/SIGTERM + crash handling so nothing is lost on exit
    setup_global_error_handler(logger)

    # 5. Fire off a worker thread that logs like normal, synchronous code
    def worker_job():
        logger.write(
            event={
                "process": "fetch_prices",
                "type": "System Process",
                "parameters": "symbol=AAPL",
                "thread_id": threading.get_ident(),
            },
            scenario="success",
            details="Fetched 200 rows",
        )

    threading.Thread(target=worker_job, daemon=True).start()

    # Keep the app alive — Logryn handles Ctrl+C / SIGTERM gracefully
    await asyncio.Event().wait()


if __name__ == "__main__":
    asyncio.run(main())
```

That's it — `write()` is safe to call from any thread, at any time, and the queue takes care of buffering and batching behind the scenes.

---

## Under the Hood (Advanced Usage)

For pipelines that track many concurrent "identities" (stations, streams, devices, workers — anything with its own config), Logryn ships `ThreadConfigManager`:

- Keeps a fast, thread-safe, in-RAM cache of config rows (by ID *and* by a custom unique key tuple), backed by a real DB table.
- Supports bootstrapping and syncing config from a **CSV file** — point it at a CSV and it validates the schema against your table, upserts the rows, and keeps a synced export on disk.
- Designed for pipelines where new "threads" of work register themselves dynamically at runtime (`add_record`) without needing a restart.

```python
from logryn import ThreadConfigManager

tcm = ThreadConfigManager(
    table_name="jobs",
    database_name="my_database",
    db_config=db_config,
    create_query="CREATE TABLE IF NOT EXISTS stations (...)",
    id_column_name="thread_id",
    unique_key_columns=("station_name",),
    csv_path="./stations.csv",
)
tcm.initialize()
```

This is optional — most users will only ever touch `MainLog` and `setup_global_error_handler`.

---

## Contributing

Contributions, bug reports, and feature requests are welcome! Feel free to open an issue or submit a pull request.

1. Fork the repo
2. Create your feature branch (`git checkout -b feature/amazing-thing`)
3. Commit your changes
4. Push and open a PR

## License

Distributed under the MIT License. See `LICENSE` for more information.
