Metadata-Version: 2.4
Name: blocksnoop
Version: 0.11.1
Summary: Detect blocking calls in asyncio event loops using eBPF + Austin
Author: Paul Milesi
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/PaulM5406/blocksnoop
Project-URL: Repository, https://github.com/PaulM5406/blocksnoop
Project-URL: Issues, https://github.com/PaulM5406/blocksnoop/issues
Keywords: asyncio,blocking,ebpf,event-loop,profiling,debugging
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: austin-python
Provides-Extra: dev
Requires-Dist: ruff<0.16,>=0.15; extra == "dev"
Requires-Dist: ty; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: docker>=7.0; extra == "dev"
Dynamic: license-file

# blocksnoop

Detect blocking calls in Python asyncio event loops using eBPF + Austin.

blocksnoop attaches to a running Python process (or launches one) and reports every time the event loop is blocked longer than a configurable threshold — with the Python stack trace that caused it.

## How it works

```
eBPF (kernel)          Austin (userspace)
  │ monitors               │ samples Python
  │ epoll gaps              │ stacks continuously
  └──────────┐   ┌─────────┘
             ▼   ▼
          Correlator
             │
             ▼
       Reporter → sinks (console, JSON, file)
```

1. An **eBPF probe** hooks the `epoll` syscalls (`epoll_wait`, `epoll_pwait`, `epoll_pwait2` — all of the family available on the kernel) and measures the time between returns (callback start) and the next entry (callback end). If the gap exceeds the threshold, it emits an event. Tracing every variant matters because which one a loop enters depends on its libc and implementation — glibc routes `epoll_wait()` through the `epoll_pwait` syscall, and uvloop/libuv call `epoll_pwait` directly.
2. A **stack sampler** ([Austin](https://github.com/P403n1x87/austin)) runs as a long-lived subprocess, continuously streaming Python stack traces into a ring buffer. Austin's pipe mode avoids per-sample subprocess overhead, enabling sub-10ms threshold detection.
3. The **correlator** enriches each blocking event with the closest matching Python stack.
4. The **reporter** fans out events to one or more output sinks.

## Requirements

- Linux with eBPF support (the default Core backend requires kernel 5.7+ for
  PID-namespace filtering and a readable epoll syscall tracepoint pair)
- Root privileges (for eBPF and Austin)
- Native `linux/amd64` or `linux/arm64` Blocksnoop wheel, or the official
  Docker image (contains the default Core sidecar)
- [BCC (BPF Compiler Collection)](https://github.com/iovisor/bcc) only for
  the explicit legacy `--backend bcc` compatibility path
- [Austin](https://github.com/P403n1x87/austin)
- [austin-python](https://github.com/P403n1x87/austin-python) (installed automatically as a dependency)
- Python 3.12+

## Installation

```bash
pip install blocksnoop
```

Or for development:

```bash
git clone git@github.com:PaulM5406/blocksnoop.git
cd blocksnoop
uv sync --all-extras --dev
```

### eBPF backends

Core is the default backend. Python launches the small `blocksnoop-ebpf`
libbpf sidecar and consumes its versioned NDJSON event stream. Protocol v2
resolves the target PID namespace and its local PID/TID before attaching, so a
hostPID collector can monitor a process in a private container namespace
without broadening its filter.

The current program attaches only to syscall tracepoints and does not read
kernel structures. Its object contains BTF metadata, but it has no
kernel-layout relocations: a readable kernel BTF file is therefore useful
diagnostic information, not a Core prerequisite. A real Core attach is the
authoritative compatibility check.

On `linux/amd64` and `linux/arm64`, PyPI selects a native wheel containing the
sidecar and precompiled object, so `pip install blocksnoop` is sufficient for
Core itself; Austin and the required privileges are still host prerequisites.
Other platforms receive the portable compatibility wheel without native
assets. A source checkout or unpacked source distribution can build the
sidecar on Linux with `make -C native` and place `native/blocksnoop-ebpf` on
`PATH`.

The official Docker image is intentionally **Core-only**: it installs the
native wheel and Austin, but not BCC or kernel headers. There is no automatic
fallback. Use `--backend bcc` only on a legacy host where you provisioned BCC
yourself.

Before attaching, inspect the exact environment and optional target without
loading BPF or spawning the sidecar:

```bash
blocksnoop doctor
blocksnoop doctor 1234
blocksnoop doctor 1234 --json
blocksnoop doctor --stats 1234  # eBPF-only pre-flight; Austin is not required
```

`doctor` exits non-zero when a required check fails and includes remediation in
both human-readable and machine-readable output. Capture mode also checks
Austin; `doctor --stats` intentionally does not. Target-specific output shows
the host and namespace-local PID/TID plus whether the collector shares the PID
namespace.

### Legacy BCC compatibility

BCC is retained for hosts that cannot run the supported Core path, such as
older kernels. It is never selected automatically and is not installed by
`pip install blocksnoop` or the official image. After provisioning BCC and
matching kernel headers yourself, select it explicitly:

```bash
sudo blocksnoop --backend bcc <PID>
sudo blocksnoop doctor --backend bcc <PID>
```

BCC is frozen as a compatibility backend through the pre-1.0 releases and any
future 1.x series; new features target Core. Its removal would not happen
before a future major release.

## Usage

### Attach to a running process

```bash
sudo blocksnoop <PID>
sudo blocksnoop -t 50 <PID>          # 50ms threshold (default: 100ms)
sudo blocksnoop --tid 1234 <PID>     # monitor specific thread
sudo blocksnoop -v <PID>             # enable debug logging
```

### Launch and monitor a process

```bash
sudo blocksnoop -- python app.py
sudo blocksnoop -t 50 -- python app.py
```

### Output modes

```bash
# Human-readable to stderr (default)
sudo blocksnoop -- python app.py

# Versioned NDJSON lifecycle to stdout (for piping to jq, etc.)
sudo blocksnoop --json -- python app.py

# Suppress individual event records; keep the start and final summary
sudo blocksnoop --json --summary-only -- python app.py

# Structured JSON to file (for Datadog/Fluentd/CloudWatch)
sudo blocksnoop --log-file /var/log/blocksnoop/events.json --service my-api --env production -- python app.py

# Combine: console to terminal + JSON to file
sudo blocksnoop --log-file /var/log/blocksnoop/events.json --service my-api -- python app.py
```

### Stats mode

Use `--stats` to run **only the eBPF detector** (no Austin profiler, no stack traces) and see the distribution of all epoll gaps. This helps you pick the right `--threshold` before running a full profiling session.

```bash
# Capture all epoll gaps and display live statistics
sudo blocksnoop --stats <PID>

# Backward-compatible JSON stats snapshots (one record per second)
sudo blocksnoop --stats --json <PID>

# Only gaps above 10ms
sudo blocksnoop --stats -t 10 <PID>
```

Unlike normal capture mode, `--stats --json` is a stream of independent,
backward-compatible statistics snapshots rather than a
`blocksnoop.events/v1` session lifecycle. Each line contains `pid`,
`elapsed_s`, `count`, `rate`, and percentile fields once events exist.

Sample output (redrawn in place every second):

```
blocksnoop stats — PID 1234 — 12.3s — 4821 events (391/s)

  min          0.0ms
  avg          2.1ms
  p50          0.8ms
  p90          4.2ms
  p95          8.7ms
  p99         45.3ms
  max        302.1ms
```

### Example output

Human-readable:

```
[   1.23s] #1   BLOCKED     302.1ms  tid=1234
  Python stack (most recent call last):
    app.py:7 in blocking_io
      time.sleep(0.5)
    app.py:13 in main
      blocking_io()

[   2.05s] #2   BLOCKED     298.5ms  tid=1234
  Python stack (most recent call last):
    app.py:7 in blocking_io
      time.sleep(0.5)
    app.py:13 in main
      blocking_io()

--- blocksnoop session ---
Duration: 8.0s
Blocking events detected: 2
Lost detector events: 0
```

Normal capture JSON (`--json` without `--stats`) is a versioned NDJSON
lifecycle. Every line is independently
parseable and shares `schema`, `schema_version`, `type`, and `session_id`.
The three record types are `session_start`, `blocking_event`, and
`session_summary`; filter events with `jq 'select(.type == "blocking_event")'`.
The final summary makes a clean zero-event session distinguishable from a
truncated stream and includes loss counts plus the top blocking call sites.

```json
{"schema":"blocksnoop.events/v1","schema_version":1,"type":"session_start","session_id":"...","backend":"core","threshold_ms":100.0,"target_pid":5678,"target_tid":5678}
{"schema":"blocksnoop.events/v1","schema_version":1,"type":"blocking_event","session_id":"...","event_number":1,"timestamp_s":1.23,"duration_ms":302.1,"pid":5678,"tid":5678,"python_stacks":[[{"function":"blocking_io","file":"app.py","line":7,"source":"time.sleep(0.5)"}]],"level":"warning"}
{"schema":"blocksnoop.events/v1","schema_version":1,"type":"session_summary","session_id":"...","termination_reason":"clean","status":"completed","duration_s":8.0,"event_count":1,"lost_event_count":0,"total_blocked_ms":302.1,"max_blocked_ms":302.1,"top_signatures":[{"location":"app.py:7 in blocking_io","count":1,"total_blocked_ms":302.1,"max_blocked_ms":302.1}]}
```

### CLI reference

```
blocksnoop [OPTIONS] [PID] [-- COMMAND ...]
blocksnoop doctor [OPTIONS] [PID]

Options:
  -t, --threshold FLOAT        Blocking threshold in ms (default: 100, or 0 with --stats)
  --stats                      eBPF-only mode: show epoll gap distribution (no Austin/stacks)
  --tid INT                    Thread ID to monitor (default: main thread)
  --backend {core,bcc}         eBPF backend to use (default: core; bcc is legacy)
  --json                       JSON lines output to stdout
  --summary-only               Suppress individual events; keep the session lifecycle
  --log-file PATH              Write structured JSON to file for log aggregators
  --service NAME               Service name for structured logs (default: blocksnoop)
  --env ENV                    Environment tag for structured logs
  --no-color                   Disable ANSI colors in terminal output
  -v, --verbose                Enable debug logging to stderr
  --error-threshold MS         Duration in ms above which events are errors (default: 500)
  --correlation-padding MS     Correlation time window padding in ms (default: 200)
  --fail-on {none,event,error} Exit 3 when a completed session violates the policy
  --fail-on-loss               Exit 3 when detector events were lost
```

## Docker

blocksnoop requires kernel access, so Docker containers need `--privileged` and `--pid=host`:

```bash
# Pull from Docker Hub
docker pull oloapm/blocksnoop

# Check that the host kernel can run the Core backend
docker run --rm --privileged --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  oloapm/blocksnoop blocksnoop doctor --json

# Attach to a process on the host with the precompiled libbpf backend
docker run --rm --privileged --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  oloapm/blocksnoop blocksnoop -t 100 <PID>

# Launch and monitor a process
docker run --rm --privileged --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  oloapm/blocksnoop blocksnoop -t 100 -- python app.py
```

For local development:

```yaml
# docker-compose.yml
services:
  blocksnoop:
    build: .
    privileged: true
    pid: host
```

```bash
docker compose run --rm blocksnoop blocksnoop -t 100 -- python app.py
```

The image is published for `linux/amd64` and `linux/arm64`. It has no BCC
fallback: a Core prerequisite failure is reported directly. For a local image
smoke before attaching to a workload:

```bash
docker build -t blocksnoop:local .
docker run --rm blocksnoop:local python -c '
from blocksnoop.core_backend import find_sidecar
assert find_sidecar()
'
docker run --rm blocksnoop:local sh -ec '
  austin --version >/dev/null
  python -c "import importlib.util; assert importlib.util.find_spec(\"bcc\") is None"
'
```

## Kubernetes

blocksnoop uses eBPF which operates at the kernel level, so you run it on the **node**, not inside the application container. The target process just needs to be visible from the host PID namespace.

> **Note:** On kernel 5.7+, both backends translate a host-visible target into
> its container-local PID/TID. A node-level collector still needs
> `hostPID: true` to see the target in `/proc`; an ephemeral container sharing the target
> process namespace can use its local PID directly.

### Ephemeral debug container (recommended)

Attach directly to a running pod with an ephemeral container. `--profile=sysadmin` (K8s 1.28+) grants the privileged access required for eBPF:

```bash
# Find the pod
kubectl get pods -l app=my-api

# Attach an ephemeral debug container with eBPF privileges
kubectl debug -it my-api-pod-7b8c9d \
  --image=oloapm/blocksnoop:latest \
  --target=my-api \
  --profile=sysadmin \
  -- sh -c "mount -t debugfs debugfs /sys/kernel/debug 2>/dev/null; exec sh"
```

> `--target` shares the process namespace with the app container, so you can see its PIDs. `--profile=sysadmin` enables privileged mode for eBPF and debugfs access.

Inside the debug container, find the Python process and attach:

```bash
# Find the Python PID
ps aux | grep python

# Attach blocksnoop
blocksnoop -t 50 <PID>

# Or with structured logging
blocksnoop --json -t 50 <PID>
```

The Core image never compiles BPF at runtime and needs neither kernel headers
nor BCC. If `blocksnoop doctor` fails, fix the reported host-kernel, tracefs,
privilege, or target-visibility prerequisite instead of adding headers to the
image.

### Cross-container attach (different mount namespaces)

When blocksnoop runs from a sidecar, ephemeral debug container, or a privileged Job pinned to the target's node, it lives in a *different mount namespace* than the target — so the Python binary paths Austin reads from `/proc/<pid>/maps` (e.g. `/usr/local/bin/python3.11`) don't exist in blocksnoop's filesystem. Without help, Austin then logs `🔢 Cannot determine the version of the Python interpreter.` and produces zero samples.

blocksnoop handles this automatically: when the target's mount namespace differs from blocksnoop's, it generates a thin wrapper that opens `austin` (and the musl linker) as file descriptors in its own namespace, then `nsenter`s into the target and execs via `/proc/self/fd/N`. fds survive `execve`, so Austin loads from blocksnoop's rootfs while sampling against the target's filesystem view.

This means:

- The target image is **never** modified — no binaries copied, no files written under `/proc/<TARGET>/root/`.
- Works against hardened targets with `readOnlyRootFilesystem: true`.
- Works regardless of the target's libc (alpine/musl, debian/glibc, distroless).
- Requires `CAP_SYS_ADMIN` on the blocksnoop side (already granted by `--profile=sysadmin` or `privileged: true`).

See `examples/reproduce-cross-ns.sh` for a runnable Docker reproduction.

### DaemonSet sidecar

For continuous monitoring, deploy blocksnoop as a DaemonSet that monitors processes on each node:

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: blocksnoop
spec:
  selector:
    matchLabels:
      app: blocksnoop
  template:
    metadata:
      labels:
        app: blocksnoop
    spec:
      hostPID: true
      containers:
        - name: blocksnoop
          image: oloapm/blocksnoop:latest
          # A DaemonSet needs an explicit target-selection policy. Set this
          # value from your workload controller; blocksnoop monitors one PID.
          env:
            - name: TARGET_PID
              value: "1234"
          command:
            - sh
            - -ec
            - >-
              exec blocksnoop --json
              --log-file /var/log/blocksnoop/events.json
              --service my-api --env production -t 100 "$TARGET_PID"
          securityContext:
            privileged: true
          volumeMounts:
            - name: logs
              mountPath: /var/log/blocksnoop
            - name: debugfs
              mountPath: /sys/kernel/debug
      volumes:
        - name: logs
          hostPath:
            path: /var/log/blocksnoop
        - name: debugfs
          hostPath:
            path: /sys/kernel/debug
```

The log file at `/var/log/blocksnoop/events.json` can be tailed by Datadog Agent, Fluentd, or any log collector running on the node.

### Node shell (quick one-off)

For a quick check without building images:

```bash
# SSH into the node (or use a node shell tool)
kubectl node-shell <node-name>

# Install blocksnoop
pip install blocksnoop

# Find the Python process (hostPID shows all processes)
ps aux | grep python

# Attach
blocksnoop -t 50 <PID>
```

## Development

```bash
# Install dependencies
uv sync --all-extras --dev

# Run unit tests
uv run --extra dev pytest tests/ -v --ignore=tests/integration

# Run integration tests (requires Docker)
uv run --extra dev pytest -m docker tests/integration/ -v

# Lint and format
ruff check blocksnoop/ tests/
ruff format blocksnoop/ tests/

# Type check
ty check blocksnoop/
```

## License

GPL-3.0-or-later (due to the [austin-python](https://github.com/P403n1x87/austin-python) dependency)
