Metadata-Version: 2.4
Name: pfio
Version: 2.10.0
Summary: PFN IO library
Author-email: Tianqi Xu <tianqi@preferred.jp>, Kota Uenishi <kota@preferred.jp>
License-Expression: MIT
Project-URL: Homepage, https://github.com/pfnet/pfio
Keywords: filesystem,hdfs,chainer,development
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3
Requires-Dist: deprecation
Requires-Dist: urllib3
Provides-Extra: test
Requires-Dist: tox>=4; extra == "test"
Requires-Dist: pytest; extra == "test"
Requires-Dist: flake8; extra == "test"
Requires-Dist: autopep8; extra == "test"
Requires-Dist: parameterized; extra == "test"
Requires-Dist: isort; extra == "test"
Requires-Dist: moto[server]>=5.0.0; extra == "test"
Requires-Dist: numpy; extra == "test"
Requires-Dist: mypy; extra == "test"
Provides-Extra: doc
Requires-Dist: sphinx; extra == "doc"
Requires-Dist: sphinx_rtd_theme; extra == "doc"
Provides-Extra: bench
Requires-Dist: numpy>=1.19.5; extra == "bench"
Requires-Dist: torch>=1.9.0; extra == "bench"
Requires-Dist: Pillow<=8.2.0; extra == "bench"
Provides-Extra: hdfs
Requires-Dist: pyarrow>=6.0.0; extra == "hdfs"
Provides-Extra: trace
Requires-Dist: pytorch-pfn-extras; extra == "trace"
Dynamic: license-file

## PFIO

![Doc Status Badge](https://readthedocs.org/projects/pfio/badge/?version=master&style=flat)

PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports

- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.


## Dependency

- HDFS client and libhdfs for HDFS access
- CPython >= 3.10

## Installation and Document build

Installation

```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```

Documentation
```sh
$ cd pfio/docs
$ uv run --extra doc make html
$ open build/html/index.html
```

## How to use

Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.

## Release

Check [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.

Run tests locally:

```sh
$ uv sync
$ uv run tox --skip-env doc
```

Bump version numbers in `pfio/version.py` .

Push and open a pull request to invoke CI. Once CI passed and the pull request merged,
tag a release:

```sh
$ git tag -s X.Y.Z
$ git push --tags
```

Build:

```sh
$ rm -rf dist
$ uv build
```

Release to PyPI:

```sh
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
```
