Metadata-Version: 2.3
Name: surf-archiver
Version: 0.1.2a6
Summary: 
Author: Isaac Williams
Author-email: i.williams@amolf.nl
Requires-Python: >=3.9,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: aio-pika (>=9.4.1,<10.0.0)
Requires-Dist: pydantic (>=2.7.1,<3.0.0)
Requires-Dist: pydantic-settings[yaml] (>=2.2.1,<3.0.0)
Requires-Dist: s3fs (>=2024.3.1,<2025.0.0)
Requires-Dist: typer (>=0.12.3,<0.13.0)
Description-Content-Type: text/markdown

# surf-archiver

The `surf-archiver` tool consists of two components: 
1) a CLI tool
2) a remote client which can execute CLI installed on a remote machine

The CLI tool copies daily data from S3 supported storage, bundling it into a per 
experiment per day tar archives. Once completed it emits a message via RabbitMQ. The 
intended use case is for the CLI tool to installed on the Surf Data Archive and remotely 
triggered daily.


The tool copies data based on the S3 key. It is assumed that the keys have the following
structure:
```
<SRC BUCKET>/<images|videos>/<EXPERIMENT-ID>/YYYYMMDD/*.tar
```

The resulting archive will be created at:
```
<DATA DIR>/<images|videos>/<EXPERIMENT-ID>/YYYY-MM-DD.tar
```

# Installation of the CLI tool

The CLI tool can be installed via [pipx](https://github.com/pypa/pipx).


In order to run the tool needs some configuration. It can be configured by via a yaml 
file:

```yaml
bucket:             # S3 bucket data is pulled from
target_dir:         # Target directory data is stored in       
connction_url:      # RabbitMQ connection url
exchange_name:      # RabbitMQ Exchange name
log_file:           # log file path
```

By default the tool will look for this configuration in the `${HOME}/.surf-archiver`.
The tool will additionally look for these parameters in the shell environment. If they
are defined here, they need to be prefixed by `surf_archiver_`. 


In addition to the above, the following environment variables need to be set in order to
connect to S3.
```bash
AWS_SECRET_KEY_ID=
AWS_ACCESS_KEY_ID=
AWS_ENDPOINT_URL=
```

