Metadata-Version: 2.4
Name: scigantic-transfer
Version: 0.1.0
Summary: Mirror a Hugging Face dataset into a region-local S3 bucket and load it into memory fast (with HF fallback + benchmark).
Author: Scigantic
License: Proprietary — Evaluation Use Only. See LICENSE.
Keywords: huggingface,s3,datasets,gpu,mirror,transfer,benchmark
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26
Requires-Dist: huggingface_hub>=0.20
Provides-Extra: fast
Requires-Dist: hf_transfer>=0.1.5; extra == "fast"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# scigantic-transfer

Mirror a Hugging Face dataset into a region-local S3 bucket, then load it into
memory fast on a GPU instance in that region.

## Install

```
pip install scigantic-transfer
```

## Mirror a dataset (Hugging Face to S3)

```
scigantic-transfer --region REGION --name HF_DATASET --bucket-name BUCKET [--include GLOB]
```

Flags:

```
--region        AWS region of the bucket and the GPU instance (must match)
--name          Hugging Face dataset id
--bucket-name   destination S3 bucket; data lands at s3://BUCKET/NAME/
--include GLOB  files to mirror, repeatable; default '*' (whole repo)
```

Examples:

```
scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror
scigantic-transfer --region us-east-1 --name org/ds --bucket-name my-mirror --include 'data/*'
```

## Load into memory

```python
from scigantic_transfer import load

result = load("org/ds", region="us-east-1", bucket="my-mirror")
data = result.blob
```

## Benchmark

```
scigantic-transfer-bench --size-gib 1
scigantic-transfer-bench --live --region us-east-1 --name org/ds --bucket-name my-mirror
```

## License

Proprietary, evaluation use only. No redistribution or derivative works.
See LICENSE. Contact support@scigantic.com.
