Metadata-Version: 2.4
Name: multiprocess-rotate-logging
Version: 0.1
Summary: A multi-process-safe replacement for Python's TimedRotatingFileHandler
Author-email: Ben Lin <benincampus@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/BenLin0/multiprocess-rotate-logging
Keywords: logging,handler,rotation,multiprocess,timed
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Logging
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# MultiProcessRotateLogging

A drop-in replacement for `TimedRotatingFileHandler` that is safe when several processes write to the same file. Same constructor arguments, same defaults — only the rotation behaviour is fixed.

## The problem

`logging.handlers.TimedRotatingFileHandler` is not multi-process-safe. At the rotation boundary:

1. **Process A** renames `app.log` to `app.log.2026-08-10`, then opens a fresh `app.log` and carries on.
2. **Process B** reaches the same boundary a moment later, sees that `app.log.2026-08-10` already exists, and bails out of the rollover:

   ```python
   if os.path.exists(dfn):
       # Already rolled over.
       return
   ```

   B returns *without touching its stream* — which is still the file A renamed. Every record B writes from then on goes into `app.log.2026-08-10`, growing yesterday's log forever while `app.log` is missing B's output entirely.

The same thing happens with external rotation (`logrotate`, `newsyslog`) or when someone deletes the log file: the handler keeps writing to a file nobody is reading.

## How it works

`MultiProcessRotateFileHandler` fixes both halves of the problem:

- **On rollover** — the process that loses the race does not keep its old stream. It sees that the dated file already exists, closes its stream, and reopens the current `app.log`, creating it if the winning process has not opened it yet.
- **On every record** — the handler compares the device and inode of its open stream against `app.log` on disk, the way `WatchedFileHandler` does. If they differ, the file was rotated or removed out from under it, and it reattaches before writing.

Only one process ever renames the file. The claim is made with an atomic hard link, which fails if the dated file already exists, so two processes rotating in the same instant can never overwrite each other's file.

```
TimedRotatingFileHandler                MultiProcessRotateFileHandler
────────────────────────                ─────────────────────────────
app.log             A:after             app.log             A:after
                                                            B:after
app.log.2026-08-10  A:before            app.log.2026-08-10  A:before
                    B:before                                B:before
                    B:after   ← lost from app.log
```

## Installation

### Via pip (recommended)

```bash
pip install multiprocess-rotate-logging
```

### Manual

No dependencies outside the standard library. Copy `multiprocess_rotate_logging.py` directly into your project.

Requires Python 3.9+.

## Usage

```python
import logging
import os
from multiprocess_rotate_logging import MultiProcessRotateFileHandler

handler = MultiProcessRotateFileHandler("app.log", when="midnight", backupCount=7)
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-8s %(name)s: %(message)s"))

logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

logger.info("Worker started, pid=%d", os.getpid())
```

The arguments are exactly `TimedRotatingFileHandler`'s, so migrating is a one-line change:

```diff
-from logging.handlers import TimedRotatingFileHandler
-handler = TimedRotatingFileHandler("app.log", when="midnight", backupCount=7)
+from multiprocess_rotate_logging import MultiProcessRotateFileHandler
+handler = MultiProcessRotateFileHandler("app.log", when="midnight", backupCount=7)
```

Every process in the application creates its own handler on the same path — no coordination, no lock files, no separate log per PID. They rotate cooperatively: the first process past midnight renames the file, and the rest notice and follow it to the new one.

### Rotating at a different time of day

Servers are often busiest at midnight. `atTime` moves the rollover:

```python
from datetime import time

handler = MultiProcessRotateFileHandler("app.log", when="midnight", backupCount=7, atTime=time(3, 30))
```

### With `logging.config.dictConfig`

```python
LOGGING = {
    "version": 1,
    "handlers": {
        "file": {
            "class": "multiprocess_rotate_logging.MultiProcessRotateFileHandler",
            "filename": "/var/log/myapp/app.log",
            "when": "midnight",
            "backupCount": 14,
            "formatter": "standard",
        },
    },
    "formatters": {
        "standard": {"format": "%(asctime)s %(levelname)-8s %(name)s: %(message)s"},
    },
    "root": {"level": "INFO", "handlers": ["file"]},
}
```

## API

### `MultiProcessRotateFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False, atTime=None, errors=None)`

The signature is identical to `TimedRotatingFileHandler`'s, defaults included — a test asserts the two match, so it stays that way. Every argument means what it does in the standard library:

| Parameter | Type | Default | Description |
|---|---|---|---|
| `filename` | `str` | — | Path to the log file |
| `when` | `str` | `'h'` | Rotation unit: `'S'`, `'M'`, `'H'`, `'D'`, `'midnight'`, or `'W0'`–`'W6'` |
| `interval` | `int` | `1` | Number of `when` units between rotations |
| `backupCount` | `int` | `0` | Dated files to keep; `0` keeps them all |
| `encoding` | `str` | `None` | File encoding; `None` means the platform default |
| `delay` | `bool` | `False` | Defer opening the file until the first record |
| `utc` | `bool` | `False` | Use UTC rather than local time for the rollover and the file suffix |
| `atTime` | `datetime.time` | `None` | Time of day to rotate at, for `midnight` and `W*` |
| `errors` | `str` | `None` | Encoding error handler, as passed to `open()` |

> **Note the default is hourly, not daily** — that is `TimedRotatingFileHandler`'s default and this handler matches it. Pass `when="midnight"` for daily rotation.

The dated suffix follows the parent's rules: `<filename>.<YYYY-MM-DD>` for daily and weekly rotation, `<filename>.<YYYY-MM-DD_HH-MM-SS>` for sub-daily, named for the interval that just ended.

Two methods are available if you need them directly:

- `reopenIfNeeded()` — reattach to the base file if it has been renamed or removed. Called automatically before every record.
- `doRollover()` — perform (or defer to another process's) rotation. Called automatically at the boundary.

## Comparison to the alternatives

| | `TimedRotatingFileHandler` | `WatchedFileHandler` | `MultiProcessRotateFileHandler` |
|---|---|---|---|
| Rotates on a schedule | Yes | No — relies on an external tool | Yes |
| Survives rotation by another process | **No** — keeps writing to the renamed file | Yes | Yes |
| Survives external rotation (`logrotate`) | No | Yes | Yes |
| Two processes can't clobber each other's dated file | No | N/A | Yes — atomic hard-link claim |
| Prunes old files (`backupCount`) | Yes | N/A | Yes, race-tolerant |

The common workaround is to point every process at `WatchedFileHandler` and let `logrotate` do the rotating. That works, but it moves your rotation policy out of the application and into system configuration. This handler keeps it in Python, next to the rest of your logging setup.

## Notes and limitations

- **Boundary records.** A record already being written when another process performs the rename lands in the dated file rather than the new one. Nothing is lost — every record is written exactly once — but timestamps at the boundary may span two files by a few milliseconds.
- **`rotator`.** The inherited `namer` hook is used when building the dated filename. The `rotator` hook is only used on filesystems without hard-link support, since the atomic claim replaces the rename.
- **Windows.** Rotation requires renaming an open file, which Windows does not allow while any process holds it open. Like the standard handler, this one is intended for POSIX systems.

## Running the demo

```bash
python3 demo.py
```

A minimal script showing the everyday configuration: rotate at midnight, keep a week, write a couple of records to `app.log`.

The multi-process behaviour is exercised by the test suite rather than the demo — `test_concurrent_processes_lose_no_records` spawns four processes that all hit the rotation boundary at the same instant and asserts that every record survives.

## Running the tests

```bash
python3 -m unittest test_multiprocess_rotate_logging -v
```
