Metadata-Version: 2.4
Name: pyetwkit
Version: 3.2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Logging
Classifier: Typing :: Typed
Requires-Dist: click>=8.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: gradio>=4.0 ; extra == 'dashboard'
Requires-Dist: pytest>=7.0 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21 ; extra == 'dev'
Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
Requires-Dist: black>=23.0 ; extra == 'dev'
Requires-Dist: ruff>=0.1 ; extra == 'dev'
Requires-Dist: mypy>=1.0 ; extra == 'dev'
Requires-Dist: sphinx>=7.0 ; extra == 'docs'
Requires-Dist: sphinx-rtd-theme>=2.0 ; extra == 'docs'
Requires-Dist: sphinx-autodoc-typehints>=2.0 ; extra == 'docs'
Requires-Dist: myst-parser>=3.0 ; extra == 'docs'
Requires-Dist: pandas>=2.0 ; extra == 'export'
Requires-Dist: pyarrow>=14.0 ; extra == 'export'
Provides-Extra: dashboard
Provides-Extra: dev
Provides-Extra: docs
Provides-Extra: export
Summary: High-performance ETW (Event Tracing for Windows) consumer library for Python
Keywords: etw,windows,tracing,events,monitoring,logging
Author: m96-chan
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/m96-chan/PyETWkit/releases
Project-URL: Documentation, https://github.com/m96-chan/PyETWkit#readme
Project-URL: Homepage, https://github.com/m96-chan/PyETWkit
Project-URL: Issues, https://github.com/m96-chan/PyETWkit/issues
Project-URL: Repository, https://github.com/m96-chan/PyETWkit

# PyETWkit

[![PyPI version](https://badge.fury.io/py/pyetwkit.svg)](https://badge.fury.io/py/pyetwkit)
[![Python](https://img.shields.io/pypi/pyversions/pyetwkit.svg)](https://pypi.org/project/pyetwkit/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/m96-chan/PyETWkit/actions/workflows/ci.yml/badge.svg)](https://github.com/m96-chan/PyETWkit/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/m96-chan/PyETWkit/branch/main/graph/badge.svg)](https://codecov.io/gh/m96-chan/PyETWkit)

A modern, high-performance ETW (Event Tracing for Windows) toolkit for Python, powered by a Rust backend.

---

## Features

### Core
- **Real-time ETW streaming** with sync API
- **Kernel providers**: process, thread, registry, file, disk, network
- **User providers**: DNS, Audio, and more via profiles
- **ETL file reading**: Parse existing trace logs
- **Full property decoding** via TDH: arrays, nested structures, SIDs, and WPP
  events given a PDB or `.tmf` — see [Event Properties](docs/advanced/event_properties.rst)
- **Rust backend (pyo3)**: High throughput, zero-copy event delivery
- Windows 10 / 11 / Server supported

### v2.0 - Enterprise Features
- **Multi-session support**: Run multiple ETW sessions simultaneously
- **Manifest-based typed events**: Parse ETW manifests for structured event data
- **Rust-side filtering**: High-performance event filtering in Rust
- **Provider discovery**: Search and list available providers
- **Pre-configured profiles**: Audio, network, security scenarios

### v3.0 - Advanced Analysis
- **Live Dashboard**: Browser-based real-time visualization with Gradio
- **Event Correlation Engine**: Auto-correlate events by PID/TID/Handle
- **Recording & Replay**: Capture and replay ETW sessions (.etwpack format)
- **OpenTelemetry Exporter**: Send events to an OTLP collector over HTTP (Jaeger, Grafana, Datadog), or write spans to a file. No extra dependency

### Export Formats
- CSV, JSON, JSONL, Parquet, Arrow

---

## Installation

```bash
pip install pyetwkit

# Optional: Dashboard support
pip install pyetwkit[dashboard]

# Optional: Export to Parquet/Arrow
pip install pyetwkit[export]
```

---

## Quick Start

### CLI Usage

```bash
# List available providers
pyetwkit providers
pyetwkit providers --search Kernel

# List profiles
pyetwkit profiles

# Listen to events (requires admin)
pyetwkit listen Microsoft-Windows-DNS-Client
pyetwkit listen --profile network

# Launch live dashboard (requires admin)
pyetwkit dashboard Microsoft-Windows-Kernel-Process
pyetwkit dashboard --profile network --port 8080

# Export ETL file
pyetwkit export trace.etl -o events.csv
pyetwkit export trace.etl -o events.parquet -f parquet
```

### Python API

```python
from pyetwkit._core import EtwProvider, EtwSession

# Create session
session = EtwSession("MySession")

# Add provider
provider = EtwProvider(
    "Microsoft-Windows-DNS-Client",
    "DNS-Client"
)
provider = provider.level(4)  # Info level
session.add_provider(provider)

# Start and process events
session.start()

try:
    while True:
        event = session.next_event_timeout(1000)
        if event:
            print(f"Event {event.event_id}: {event.provider_name}")
except KeyboardInterrupt:
    pass
finally:
    session.stop()
```

### Live Dashboard

```python
from pyetwkit import Dashboard

# Create and launch dashboard
dashboard = Dashboard(port=7860)
dashboard.add_provider("Microsoft-Windows-Kernel-Process")
dashboard.add_provider("Microsoft-Windows-DNS-Client")

# Opens browser at http://localhost:7860
dashboard.launch()
```

### Event Correlation

```python
from pyetwkit import CorrelationEngine

# Create correlation engine
engine = CorrelationEngine()
engine.add_provider("Microsoft-Windows-Kernel-Process")
engine.add_provider("Microsoft-Windows-Kernel-Network")

# Add events from your ETW session
for event in events:
    engine.add_event(event)

# Correlate events by process ID
correlated = engine.correlate_by_pid(1234)
for event in correlated:
    print(f"Event {event.event_id} from {event.provider_name}")

# Export to timeline JSON
timeline = engine.to_timeline_json(pid=1234)
```

### Recording & Replay

```python
from pyetwkit import Recorder, Player, CompressionType, RecorderConfig

# Record events
config = RecorderConfig(compression=CompressionType.ZSTD)
recorder = Recorder("session.etwpack", config=config)
recorder.add_provider("Microsoft-Windows-DNS-Client")
recorder.start()

# ... capture events ...
recorder.stop()

# Replay events
player = Player("session.etwpack")
print(f"Duration: {player.duration:.2f}s, Events: {player.event_count}")

for event in player.events():
    print(f"Event {event['event_id']}")
```

### OpenTelemetry Export

Spans are sent as OTLP/HTTP with JSON encoding, so no extra dependency is
needed. Note **4318** — 4317 is the gRPC port and will not answer HTTP.

```python
from pyetwkit import OtlpExporter, SpanMapper

# Map ETW events to spans
mapper = SpanMapper()
mapper.add_rule(
    provider="Microsoft-Windows-Kernel-Process",
    event_id=1,
    span_name="process.start",
    attributes=["ProcessID", "ImageName"],
)

exporter = OtlpExporter(
    endpoint="http://collector:4318",
    service_name="my-service",
    resource_attributes={"deployment.environment": "production"},
    span_mapper=mapper,
)

for event in events:
    exporter.export(event)

# False means nothing was delivered; the batch is kept so it can be retried.
if not exporter.flush():
    log.warning("OTLP export failed; see the log for the reason")
```

To write spans to a file instead, for a collector agent to pick up:

```python
from pyetwkit import OtlpFileExporter

exporter = OtlpFileExporter("traces.json", service_name="my-service")
for event in events:
    exporter.export(event)
exporter.flush()
```

### Kernel Tracing

```python
from pyetwkit._core import PyKernelFlags, PyKernelSession

flags = PyKernelFlags()
flags = flags.with_process()  # Enable process events

session = PyKernelSession(flags)
session.start()

for _ in range(10):
    event = session.next_event_timeout(1000)
    if event and event.event_id == 1:  # Process start
        props = event.to_dict().get("properties", {})
        print(f"Process: {props.get('ImageFileName')}")

session.stop()
```

### Provider Discovery

```python
from pyetwkit._core import list_providers, search_providers

# List all providers
for p in list_providers()[:10]:
    print(f"{p.name}: {p.guid}")

# Search by name
for p in search_providers("Kernel"):
    print(p.name)
```

### Export Events

```python
from pyetwkit._core import EtlReader
from pyetwkit.export import to_csv, to_parquet

# Read ETL file
reader = EtlReader("trace.etl")
events = list(reader.events())

# Export to various formats
to_csv(events, "events.csv")
to_parquet(events, "events.parquet")
```

---

## Architecture

```
Python API / CLI
  ↓
pyetwkit (Python package)
  ↓
pyetwkit._core (Rust/pyo3)
  ↓
ferrisetw (Rust ETW library)
  ↓
Windows ETW subsystem
```

---

## Documentation

- [Tutorial](docs/tutorial.md) - Comprehensive usage guide
- [API Reference](docs/api/) - Detailed API documentation
- [Examples](examples/) - Sample scripts
- [Architecture](docs/architecture/) - Design documents
- [Event Properties](docs/advanced/event_properties.rst) - How values are decoded, display strings, raw payloads, and WPP

---

## Changelog

### v3.2.0 (2026-09)

- **OpenTelemetry export actually sends** (#90). Spans are POSTed to `/v1/traces`
  as OTLP/HTTP with JSON encoding, using only the standard library — no new
  dependency. `flush()` returns `False` and logs the reason on failure, keeping
  the batch so events can be retried rather than lost
- Fixed: the OTLP JSON encoding sent enum *names*, which a collector rejects.
  The spec allows integers only, so `kind` and `status.code` are now numbers
- Fixed: `event_to_span()` crashed on every real event. `EtwEvent.timestamp` is
  an RFC 3339 string and the code called `float()` on it; only mocks and plain
  numbers had ever been passed to it

Note the OTLP/HTTP port is **4318**. The 4317 in earlier examples is the gRPC
port and will not answer an HTTP request.

### v3.1.0 (2026-09)

Event properties are now decoded from the schema via TDH, rather than guessed
from a list of twelve names. See [Event Properties](docs/advanced/event_properties.rst).

- **All properties decoded**: whatever the provider declares, not a guess list (#72, #76)
- **Arrays and nested structures**: lists and dicts, instead of only the first element (#84)
- **WPP events**: decoded given a `.pdb` or `.tmf` — `set_wpp_pdb_path()` needs no SDK tooling (#72)
- **Undecodable events keep their payload** in `raw_data` instead of losing it
- **`formatted_properties`**: TDH's own display strings, opt-in, with value maps resolved
- Correct SIDs, `win:Boolean`, pointer widths from 32-bit processes, and counted strings
- **`pip install .` works** — it never had (#79)
- `OtlpExporter` no longer reports success while discarding events; its transport
  was never implemented (#88). `OtlpFileExporter` is unaffected

_Note: v3.0.2 was tagged and released on GitHub but never reached PyPI, so this
is the first release since v3.0.1 that PyPI users will see._

### v3.0.0 (2024-12)
- **Live Dashboard**: Gradio-based real-time UI (`pyetwkit dashboard` CLI)
- **Event Correlation Engine**: Link events by PID/TID/Handle with timeline export
- **Recording & Replay**: Capture sessions to `.etwpack` format with compression
- **OpenTelemetry Exporter**: Export to OTLP endpoints (Jaeger, Grafana, etc.) — _announced here, but the transport was not actually implemented until v3.2.0 (#88, #90)_

### v2.0.0 (2024-12)
- **Multi-session support**: Run multiple ETW sessions simultaneously
- **Manifest-based typed events**: Parse ETW provider manifests
- **Rust-side filtering**: High-performance filtering with `RustEventFilter`
- **Enhanced CLI**: Provider profiles, export options

### v1.0.0 (2024-12)
- Initial release
- Real-time ETW streaming
- Kernel and user-mode providers
- ETL file reading
- Export to CSV, JSON, JSONL, Parquet, Arrow
- CLI tool with provider discovery

---

## Examples

See the [examples/](examples/) directory for complete sample scripts:

- `basic_session.py` - Simple ETW session
- `kernel_trace.py` - Kernel-level process monitoring
- `export_events.py` - Capture and export events
- `provider_discovery.py` - Find ETW providers
- `profiles.py` - Use pre-configured profiles
- `read_etl.py` - Read ETL files
- `demo_v2_features.py` - v2.0 features demo
- `demo_v3_features.py` - v3.0 features demo

---

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

---

## License

[MIT](LICENSE)

---

## Author

[m96-chan](https://github.com/m96-chan)

