Metadata-Version: 2.4
Name: spark-mf4
Version: 2.0.0
Summary: Read ASAM MDF/MF4 measurement files into Apache Spark, backed by asammdf
Author-email: Preet Ranjan <preetish.888@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/PreetRanjan/spark-mf4
Keywords: asam,mdf,mf4,spark,databricks,can,automotive
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: asammdf>=8.0
Requires-Dist: pandas>=1.5
Requires-Dist: numpy
Provides-Extra: spark
Requires-Dist: pyspark>=3.4; extra == "spark"
Requires-Dist: pyarrow; extra == "spark"

# spark-mf4 (v2) — asammdf-backed Spark reader

**v2 changes the approach.** Instead of a hand-written MDF parser (v1, the Scala
data source), v2 delegates parsing to the mature [`asammdf`](https://github.com/danielhrisca/asammdf)
library and distributes the work across Spark. This reads the real-world files
(CANedge, Vector, ETAS, vendor variants) that a hand-written parser can silently
fail on — e.g. returning 0 rows — because `asammdf` handles the full spec.

## How it works

Spark's built-in `binaryFile` source lists and reads each file as
`(path, content)` on **any** storage it supports (DBFS, Unity Catalog Volumes,
S3, ADLS, local). The bytes are handed to `asammdf` through an in-memory buffer
inside `mapInPandas`, so there's no dependence on a local filesystem path on the
executors, and files are read in parallel (one task per file).

```
binaryFile (path, content)  ──►  mapInPandas  ──►  asammdf.MDF(BytesIO(content))
                                                     └► pandas ─► Spark rows
```

## Install (Databricks)

`pyspark` and `pyarrow` are already on the cluster. Install the library
(cluster Libraries → PyPI, or in a notebook):

```python
%pip install spark-mf4        # once published; or: %pip install /Volumes/.../spark_mf4-2.0.0-py3-none-any.whl
```

## Use

```python
from spark_mf4 import read_mf4

# A single file, a directory, or a glob — all read in parallel.
df = read_mf4(spark, "/Volumes/main/default/logs/")
df.display()

# Options
read_mf4(spark, "s3://bucket/log.mf4", raster=0.1)        # resample to 0.1 s
read_mf4(spark, ".../log.mf4", channel_group=0)            # one channel group only
read_mf4(spark, ".../log.mf4", channels=["EngineSpeed"])   # subset of channels
```

Output columns: `timestamp`, one column per channel (names sanitised for Spark),
and `source_file` (the originating path; disable with `add_source_column=False`).

### Options

| Option | Default | Meaning |
|--------|---------|---------|
| `channels` | `None` | Restrict to these channel names (whole file otherwise) |
| `channel_group` | `None` | Read only this channel group (native time base) instead of the merged wide table |
| `raster` | `None` | Resample to this many seconds |
| `time_from_zero` | `False` | Shift timestamps to start at 0 |
| `add_source_column` | `True` | Append the source file path as `source_file` |
| `path_glob` | `*.{mf4,mdf,dat}` | Which files to pick up under a directory |
| `recursive` | `True` | Recurse into sub-directories |

## Notes

- Heterogeneous files in one `read_mf4` call share the schema inferred from the
  first file; read files of the same type together.
- CAN/LIN raw payloads come through as-is; DBC signal decoding (via
  `asammdf.extract_bus_logging`) is a planned follow-up.
- v1 (the pure-Scala `format("mf4")` data source) remains on `main`; v2 lives
  here under `python/`.
