Metadata-Version: 2.4
Name: logsift-cli
Version: 0.1.0
Summary: Filter and aggregate log files through pluggable parsers.
Project-URL: Repository, https://github.com/wazeemlabs/logsift
Project-URL: Book, https://github.com/wazeemlabs/claude-code-book
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: hypothesis; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: setuptools>=64; extra == "dev"

# logsift

Read log files, parse each line into a structured record, filter and aggregate
those records. Log formats come from parser plugins that ship as ordinary
Python packages, so teaching logsift a new format never means touching logsift.

Python 3.10 or newer. No runtime dependencies.

`SPEC.md` is the contract for v1; this README is the tour.

## Install

```
pip install logsift-cli
```

Two parsers ship with it: `clf` (Apache/nginx Common and Combined Log Format)
and `logfmt` (generic `key=value` lines). Both register through the same entry
point group a third party would use.

## Quick start

```
logsift access.log --filter 'status >= 500'
logsift access.log --filter 'status >= 500' --agg 'top 10 path'
logsift app.log --filter 'level >= WARN and message ~ "timeout"' --json
logsift access.log.gz --filter 'ts > 2026-09-01T00:00:00Z' --agg 'p95 bytes'
cat app.log | logsift --parser logfmt --columns ts,level,service,message
logsift app.log --filter 'parse_error == true' --columns lineno,raw
```

Records go to stdout, diagnostics go to stderr, always, so a pipe downstream
sees a clean stream:

```
$ logsift access.log --agg 'count by status'
logsift: access.log: parser=clf (sniffed)
status  count
------  -----
200     4
204     1
...
logsift: 10 lines, 10 parsed, 0 parse errors, 0 blank, 10 matched
```

## Input

Positional arguments are paths; `-` or no argument at all means stdin. A name
ending in `.gz` is read through gzip, by suffix and never by magic bytes.

Multiple inputs are concatenated in the order given, not merged
chronologically, so two rotated files read back to back show a jump in time at
the boundary. Each input is sniffed for its own parser, so a run over a mix of
formats works.

Every line becomes a record. A line no parser understands becomes a record with
`parse_error` set and its raw text intact; nothing is dropped silently. Blank
lines are skipped and counted separately. Bytes that are not UTF-8 decode to
replacement characters instead of ending the run.

## Filtering

One flag, `--filter EXPR`. The expression is lexed and parsed by hand, so there
is no `eval` and no way for a filter to run code.

```
comparison := FIELD OP LITERAL
OP         := == != < <= > >= ~
```

Field on the left, literal on the right, always. `and`, `or`, `not` compose
them, with `not` binding tightest and parentheses overriding.

These names always mean the record's own: `ts`, `level`, `message`, `source`,
`raw`, `lineno`, `parse_error`. Anything else is looked up in whatever the
parser produced.

The literal's syntax picks the comparison:

| literal | mode | example |
| --- | --- | --- |
| `123`, `-4`, `1.5` | numeric | `status >= 500` |
| `"text"`, `'text'` | string | `path == "/health"` |
| `true`, `false` | boolean | `parse_error == true` |
| a level name | level | `level >= WARN` |
| an unquoted ISO 8601 timestamp | temporal | `ts > 2026-09-01T10:00:00Z` |

`~` is a Python regex search against the value's string form. There is no `!~`;
write `not (path ~ "/health")` and read the next section before you do.

### The one thing that surprises people

A comparison a record cannot answer is UNKNOWN, not false: the field is absent,
or `None`, or holds a type the literal cannot be compared against. UNKNOWN
propagates through `and` and `or` the way SQL's `NULL` does, and `not UNKNOWN`
is still UNKNOWN. A record prints only when the whole expression is TRUE.

So `status >= 500` and `not (status >= 500)` both exclude a line that has no
`status` field at all. That is deliberate, and it is why a typo matches
nothing rather than everything. There is no schema to check a name against, so
the stderr summary lists every field the filter named that no record carried:

```
logsift: filter fields never present on any record: staus; check for a typo
```

## Aggregation

`--agg` replaces record output; the stderr summary still prints.

```
count            count by FIELD           top N FIELD
p50 FIELD        p95 FIELD
```

Grouped output sorts by count descending then key ascending, so it is stable
across runs and diffable. A record whose group field is absent or `None` lands
in `<missing>`, which includes parse-error records: their fields are empty by
definition, and the summary's parse-error count is what explains a large
bucket.

Percentiles are exact, not estimated, which means every value is buffered at 8
bytes each; logsift warns past 10 million of them and keeps going. `count by`
holds one counter per distinct key and warns past a million groups. Both limits
are warnings, never refusals.

## Output

Table by default. Columns come from the parser, or from `--columns a,b,c` with
any name from the filter namespace. Widths are set by sampling the first 100
rows, capped at 60 characters, and longer values are truncated with `...`, so
memory stays bounded whatever the input size. `None` prints as `-`, and control
characters are escaped so one record is always one line.

`--json` prints JSONL: one flat object per line, core keys first in a fixed
order, then the parser's extras sorted by key. The shape mirrors the filter
namespace exactly, so anything you can filter on you can select with `jq` by
the same name. `raw` is always there, so the output is lossless.

`logsift big.log | head -20` exits 0 and quietly. There is no `--limit`; `head`
is the limiter.

## Time zones

Every timestamp inside logsift is timezone aware and in UTC. A parser that
returns a naive timestamp gets it localized with `--assume-tz` (default: the
system zone), and the first time that actually happens one stderr line says so,
so the assumption is never invisible. `--tz` changes display only, never
comparison. An unknown zone name exits 2.

## Exit codes

| code | meaning |
| --- | --- |
| 0 | success, including a filter that matched nothing |
| 1 | invalid `--filter` or `--agg` expression |
| 2 | missing or unreadable input, unknown or ambiguous `--parser`, no parser claimed the input, unknown timezone, plugin circuit breaker, usage error |

An empty result is a legitimate answer to a legitimate question, so it is 0.
This differs from `grep` on purpose.

## Writing a parser

Publish an object to the `logsift.parsers` entry point group. Structural
typing, so no base class and no import from logsift is required, though
`logsift.Parser` is there to type-check against and `logsift.parse_iso8601`
is there to save you writing an ISO 8601 reader.

```python
# myparser.py
from logsift import ParseResult


class SyslogParser:
    api_version = 1
    name = "syslog"
    priority = 100  # built-ins sit at 50 and below
    columns = ("ts", "level", "message")

    def can_parse(self, sample):  # up to 100 non-empty lines
        return sum(line.startswith("<") for line in sample) > len(sample) // 2

    def parse(self, line):  # ParseResult, or None for "not mine"
        ...


PARSER = SyslogParser()
```

```toml
[project.entry-points."logsift.parsers"]
syslog = "myparser:PARSER"
```

The entry point must resolve to an object, not a class, so compiled regexes
and lookup tables are built once. Installing the distribution is the whole
installation step: there is no plugin directory and no `--plugin` flag.

Your parser is contained. `parse` returning `None` or raising costs one line,
which becomes a parse-error record carrying the reason; raising from
`can_parse` costs you that one input; raising on 100 consecutive lines aborts
the run with exit 2, naming you, rather than producing ten million error
records. Under `-v` you get one traceback per exception type.

Names are addressed bare (`--parser syslog`). If two installed distributions
publish the same name, `--parser` exits 2 and lists them, `--parser DIST:NAME`
disambiguates, and sniffing picks the higher priority with a warning.

## Development

```
pip install -e ".[dev]"
pytest -q                       # full suite, including hypothesis
pytest -q -m "not slow"         # skip the wheel build and fresh-venv install
ruff check . && ruff format .
mypy --strict src/ tests/
```

Changing the entry points in `pyproject.toml` needs a reinstall before
discovery sees it; a stale `.dist-info` shows up as "no parser claimed the
input".
