Metadata-Version: 2.4
Name: shardpack
Version: 0.1.1
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Dist: pydantic>=2.10,<3
Summary: S3-first streaming format, writer, cache, and training loader for ML datasets
Keywords: machine-learning,object-storage,s3,streaming,training-data
License-Expression: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/SunDoge/shardpack/tree/main/docs
Project-URL: Issues, https://github.com/SunDoge/shardpack/issues
Project-URL: Repository, https://github.com/SunDoge/shardpack

# ShardPack Python SDK

Native Python bindings for writing and sequentially streaming immutable
ShardPack training datasets from local filesystems and S3-compatible storage.

```bash
pip install shardpack
python -m shardpack --help
```

The package provides packed-buffer dataset writing, verified sequential reads,
distributed shard assignment, bounded record shuffle, NVMe cache population,
and a framework-neutral threaded decode loader. PyTorch is not a runtime
dependency; an optional thin adapter is available when Torch is installed.

Both an inline configuration mapping and a TOML path are supported:

```python
from shardpack import Dataset

dataset = Dataset(
    "layouts/<layout-id>.layout.pb",
    config={
        "version": 1,
        "default_store": "local",
        "stores": {"local": {"type": "filesystem", "root": "/datasets"}},
    },
)
for record in dataset.records(cache_dir="/local-nvme/shardpack"):
    consume(record.sample_id, record.payload)
```

The installed `shardpack` command and `python -m shardpack` execute the same
native `usage-rs` CLI without shipping a duplicate binary. Embedded callers can
use `shardpack._native.cli_main(args)`, which returns the process-style exit code.

Development uses `uv` and `maturin`:

```bash
uv sync
uv run maturin develop
uv run pytest
```

From the repository root, the equivalent retained checks are:

```bash
mise run test:python
mise run build:python
```

The release task produces a CPython 3.11+ abi3 wheel with a manylinux 2.17
baseline. `uv run maturin sdist --out dist` produces a self-contained source
distribution containing the required Rust workspace crates.

