Metadata-Version: 2.3
Name: awesome-log-data
Version: 0.1.0
Summary: Add your description here
Author: Lucas Corcodilos
Author-email: Lucas Corcodilos <corcodilos.lucas@gmail.com>
Requires-Dist: click>=8.4.2
Requires-Dist: evtx>=0.12.1
Requires-Dist: logfire>=4.41.0
Requires-Dist: polars>=1.43.2
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pygrok>=1.0.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: xmltodict>=1.0.2
Requires-Python: >=3.14
Description-Content-Type: text/markdown

# awesome-log-data

A pipeline that ingests security log datasets from multiple sources — Windows
event logs, cloud audit trails, Linux auditd, network logs, and more — and
normalizes them into one consistent record format. The goal is a single,
straightforward way to pull structured, high-signal security log data into a
downstream project (e.g. for embedding/ML work) without hand-rolling a parser
per dataset.

Each dataset keeps its original field structure — records aren't remapped to
a shared schema — but every record is wrapped the same way and indexed the
same way, so consuming five different log formats looks the same from the
calling code's perspective.

## Datasets

| `dataset_id` | What it is | License | Labeled |
|---|---|---|---|
| `otrf` | [OTRF Security Datasets](https://github.com/OTRF/Security-Datasets) — Windows/cloud event logs (Sysmon, Windows Event Log, CloudTrail) generated by simulated attack techniques, plus raw Linux auditd logs and some Zeek network logs | GPL-3.0 | yes |
| `evtx_attack_samples` | [EVTX-ATTACK-SAMPLES](https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES) — raw Windows `.evtx` event log files, one or more per attack technique | GPL-3.0 | yes |
| `flaws_cloud` | [flaws.cloud](https://summitroute.com/downloads/flaws_cloudtrail_logs.tar) — real AWS CloudTrail logs from a public cloud-security CTF | unspecified | yes |
| `ait_lds` | [AIT Log Data Set v2](https://zenodo.org/records/19483937) — Apache access logs, Linux auditd logs, Suricata NIDS events, and Metricbeat/ECS JSON from simulated multi-host attack scenarios | CC-BY-NC-SA-4.0 | no |
| `elastic_fixtures` | [elastic/integrations](https://github.com/elastic/integrations) test fixtures — real-world sample logs for hundreds of vendor products (AWS, Okta, Auth0, Microsoft 365, CrowdStrike, and more), used by Elastic to test their own ingest pipelines | Elastic-2.0 | no |
| `splunk_attack_data` | [Splunk attack_data](https://github.com/splunk/attack_data) — Windows Event XML, Linux auditd, and cloud/SaaS JSON logs generated per MITRE ATT&CK technique | Apache-2.0 | yes |

"Labeled" means each record can be tied back to the specific attack
technique that produced it (per the manifest's `labeled` field) — useful if
you want ground truth for a specific behavior, not just a corpus of logs.

Each dataset's exact source URL and license are also recorded per-file in
`data/manifest.jsonl` after ingestion (see [Manifest](#manifest) below) —
the table above is a summary, not the source of truth.

## Requirements

- Python 3.14+
- [uv](https://docs.astral.sh/uv/)

Install dependencies:

```sh
uv sync
```

## Fetching raw data

```sh
scripts/fetch_datasets.sh list              # list available dataset_ids
scripts/fetch_datasets.sh <dataset_id>       # fetch one dataset
scripts/fetch_datasets.sh all                # fetch every dataset
```

This downloads each dataset's raw files into `data/raw/<dataset_id>/`
(gitignored — raw data isn't checked into the repo). Requires `git` and
`curl`; `ait_lds` additionally requires `jq`, and `splunk_attack_data`
additionally requires [`git-lfs`](https://git-lfs.com) (its raw samples are
stored via LFS; the fetch script pulls a scoped subset of the ~23GB upstream
repo rather than the whole thing — only files whose extension could plausibly
be a format the adapter supports).

## Ingesting a dataset

Once a dataset's raw files are on disk (via the fetch script, or placed
there yourself), run the CLI:

```sh
uv run python -m awesome_log_data.cli <dataset_id> <path-to-raw-files>
```

For example:

```sh
uv run python -m awesome_log_data.cli otrf data/raw/otrf
```

This walks the raw files, extracts archives as needed, parses every
recognized log file, and writes:

- **`data/manifest.jsonl`** — one entry per ingested source file (see
  [Manifest](#manifest)).
- **`data/parsed/<dataset_id>/`** — the parsed records themselves, sharded
  across fixed-size JSONL files with a `shard_index.parquet` (plus a small
  `source_ids.jsonl` sidecar) for random access (see
  [Reading parsed data](#reading-parsed-data)).

Re-running the same command is safe and incremental: files already recorded
in the manifest (by checksum) are skipped, so only newly added raw files get
parsed and appended.

## Record format

Each shard file holds the bare parsed record on every line, in that
dataset's native field structure — not remapped to a common schema, and not
wrapped in any envelope, so the shard files can be read directly as the
training corpus:

```json
{"eventName": "GetObject", "...": "..."}
```

Provenance for each record lives alongside it in `shard_index.parquet`, not in
the record itself (see [Reading parsed data](#reading-parsed-data)):

- **`source_id`** — an int identifying the raw file this record came from.
  It's an index into `source_ids.jsonl` (one JSON string per line, line
  number = id) rather than the string itself, to avoid repeating that string
  once per record — a source file's records are otherwise identical strings
  repeated thousands of times over. Look the string up via
  `dataset.source_ids[source_id]`, then that string (`<dataset_id>/<file_name>`,
  or `<dataset_id>/<file_name>#<hash>` if two different files happened to
  share a `file_name` — see the manifest's collision handling) in
  `data/manifest.jsonl` for the file's source URL, license, checksum, etc.
- **`record_ref`** — an offset back into the *original raw file* (byte
  offset, array index, or Windows event record ID depending on the format)
  that can re-derive this exact record from source, independent of the
  parsed/sharded copy.

## Manifest

`data/manifest.jsonl` has one entry per ingested raw file:

```json
{
  "source_id": "otrf/ec2_proxy_s3_exfiltration/ec2_proxy_s3_exfiltration_2020-09-14011940.json",
  "dataset_id": "otrf",
  "file_name": "ec2_proxy_s3_exfiltration/ec2_proxy_s3_exfiltration_2020-09-14011940.json",
  "source_url": "https://github.com/OTRF/Security-Datasets",
  "license": "GPL-3.0",
  "ingested_at": "2026-08-09",
  "checksum_sha256": "...",
  "bytes": 106600,
  "record_count": 103,
  "labeled": true,
  "record_ref_type": "byte_offset",
  "notes": ""
}
```

## Reading parsed data

Use `ShardedDataset` for indexed, random-access reads over a dataset's
already-parsed records — no need to load the whole corpus into memory:

```python
from awesome_log_data.sharded_dataset import ShardedDataset

dataset = ShardedDataset(Path("data/parsed/otrf"))

len(dataset)             # total record count
dataset[0]                # {"eventName": "GetObject", "...": "..."} - the bare parsed record
dataset.indices_for_source("otrf/ec2_proxy_s3_exfiltration/...json")  # all record indices from one raw file
```

Each record's provenance is a separate `ShardIndexEntry` at the same index in
`dataset.index` — `source_id` (an int; resolve the string via
`dataset.source_ids`, then cross-reference against `data/manifest.jsonl`) and
`record_ref` (to re-derive the record from its original raw file):

```python
from awesome_log_data.manifest import ManifestStore

manifest = ManifestStore(Path("data/manifest.jsonl"))
entry = manifest.get(dataset.source_ids[dataset.index[0].source_id])
entry.source_url, entry.license, entry.labeled
```

To re-derive a record directly from its original raw file instead of the
parsed copy (e.g. to see the exact original bytes, not just the parsed
fields), use the registered adapter's parser together with the manifest
entry's `record_ref_type` and `dataset.index[0].record_ref`:

```python
from awesome_log_data.adapters import get_adapter

adapter = get_adapter(entry.dataset_id)
source = next(
    s for s in adapter.discover(Path("data/raw/otrf"))
    if s.file_name == entry.file_name
)
source.parser.resolve(source.path, dataset.index[0].record_ref)
```

This requires the original raw files (`data/raw/<dataset_id>/`) still
present on disk — `ShardedDataset` alone is sufficient for everything else.
