Metadata-Version: 2.4
Name: fstio
Version: 0.1.1
Summary: Read R fst files natively in Python, without R or rpy2
Keywords: fst,R,dataframe,pandas,columnar
Author: 0xvsquareg
Maintainer: 0xvsquareg
License-Expression: MPL-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering
Project-URL: Documentation, https://github.com/0xvsquareg/fstio#readme
Project-URL: Issues, https://github.com/0xvsquareg/fstio/issues
Project-URL: Source, https://github.com/0xvsquareg/fstio
Requires-Python: >=3.10
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Provides-Extra: release
Requires-Dist: build>=1.2; extra == "release"
Requires-Dist: twine>=5; extra == "release"
Description-Content-Type: text/markdown

# fstio

`fstio` reads R [`fst`](https://www.fstpackage.org/) data files natively in
Python. It does not launch R and does not depend on `rpy2`.

> Who uses R anymore, right? We know. That's why `fstio` exists: so those old
> data frames are still only one line away.

The package binds the upstream `fstlib` implementation, including its LZ4 and
Zstandard codecs. Reads retain fst's column projection, row-range access, and
integrity checks.

## Install

```bash
pip install fstio
```

Binary wheels are intended for CPython 3.10–3.14 on macOS, Linux, and Windows.
Building from source requires a C/C++ compiler and CMake; it still does not
require R.

## Read a file

```python
from fstio import read_fst

frame = read_fst("data.fst")

# Project columns and read rows [1000, 2000).
subset = read_fst(
    "data.fst",
    columns=["timestamp", "price"],
    start=1000,
    stop=2000,
)
```

Unlike R's `read_fst`, `start` is zero-based and `stop` is exclusive, matching
normal Python slicing. `as_dataframe=False` returns an insertion-ordered `dict`
of arrays instead of a pandas DataFrame.

Reads use adaptive column parallelism by default. `threads="auto"` uses up to
eight workers for sufficiently large multi-column reads; set `threads=1` for
single-threaded execution or pass a positive integer to choose the limit.

```python
columns = read_fst("data.fst", as_dataframe=False, threads=4)
```

## Inspect metadata

Metadata reads only the fst headers and column names.

```python
from fstio import FstFile, metadata_fst

meta = metadata_fst("data.fst")
print(meta.num_rows, meta.column_names, meta.keys)

file = FstFile("data.fst")  # caches metadata for repeat reads
prices = file.read(columns=["price"])
```

## Type mapping

| R/fst type | Python result |
|---|---|
| integer | NumPy `int32`, or pandas nullable `Int32` when missing |
| double | NumPy `float64` |
| logical | pandas nullable Boolean array |
| character | strings with `None` for missing values |
| factor / ordered factor | pandas `Categorical` |
| `integer64` | NumPy `int64`, or pandas nullable `Int64` when missing |
| raw | NumPy `uint8` |
| `Date`, `POSIXct`, `nanotime` | pandas datetime values |
| `difftime`, `ITime` | pandas timedelta values |

The current release is read-only. The fst byte-block type is also excluded
because its reader is unimplemented in upstream `fstlib`; ordinary R `fst`
tables do not use that type.

## Benchmark

On the motivating `20260627.fst` file (6.82 MiB, 86,400 rows × 32 columns),
adaptive `fstio` is about three times faster than the stock R reader while
returning a pandas DataFrame:

| Operation | Median | P25 | P75 |
|---|---:|---:|---:|
| `fstio.read_fst` (`threads="auto"`) | 3.39 ms | 3.26 ms | 3.50 ms |
| R `fst::read_fst` (1 thread) | 11.00 ms | 11.00 ms | 12.00 ms |
| R `fst::write_fst(compress = 50)` (1 thread) | 10.00 ms | 9.25 ms | 12.00 ms |

Measured on 2026-07-19 on an Apple M4 Pro (12 logical cores), macOS 15.1,
Python 3.11.9, R 4.5.1, and R `fst` 0.9.8. Results are medians of 50 measured
runs after 10 warmups with a warm filesystem cache. The Python timing includes
metadata validation, decompression, type conversion, and DataFrame creation;
the R package installed on this macOS machine was built without OpenMP and
reported one effective thread. The R write benchmark starts with the table in
memory and produced a 7.29 MiB file.

For a separate multithreaded comparison, an OpenMP-enabled build of R
`fstcore` reached a best median of 6.13 ms with four threads on this file. On a
10× version (76.4 MiB, 864,000 rows × 32 columns), `fstio` took 29.67 ms versus
36.50 ms for the best OpenMP-enabled R run. These warm-cache figures describe
these files and this machine, not universal throughput.

Reproduce the benchmark on another file or machine with:

```bash
python benchmarks/benchmark.py data.fst --warmups 10 --repeats 50
```

The benchmark prints both R's effective thread count and whether its `fstcore`
build has OpenMP. Use `--python-threads` and `--r-threads` for controlled runs.

## Development

```bash
python -m venv .venv
. .venv/bin/activate
pip install -e '.[test,release]'
pytest
python -m build
twine check dist/*
```

The test fixture is produced by R's `fst` package but tests run without R.

## License and provenance

`fstio` and the vendored `fstlib` code are distributed under MPL-2.0. See
[`NOTICE`](NOTICE) for the pinned upstream revision and bundled codec notices.
