Metadata-Version: 2.4
Name: soreplicator
Version: 0.1.0
Summary: Simons Observatory REplicator server and command-line client
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: boto3<2,>=1.40
Requires-Dist: defusedxml<1,>=0.7.1
Requires-Dist: fastapi<1,>=0.116
Requires-Dist: httpx<1,>=0.28
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: pydantic-settings<3,>=2.10
Requires-Dist: python-multipart<1,>=0.0.20
Requires-Dist: soauth<1,>=0.8.2
Requires-Dist: socolors<1,>=0.0.8
Requires-Dist: uvicorn[standard]<1,>=0.35
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.4; extra == "dev"
Requires-Dist: pytest-asyncio<2,>=1.1; extra == "dev"
Requires-Dist: ruff<1,>=0.12; extra == "dev"

SORE: Simons Observatory REplicator
===================================

SORE is a tool to ensure that a copy of your critical data is stored on all of our
HPC machines. You can upload a file to SORE, and within an hour it should be
replicated across NERSC, Princeton, and SO:UK. The system works by leveraging
S3: we use Versity's S3 Gateway to allow you to upload data to the NERSC filesystem
from anywhere in the world, and then a background Globus job ensures data is
replicated to the other systems. You can only add and delete data that you have
uploaded; no 'edit' functionality exists. You should also not attempt to
directly modify data on disk at any of the sites (and this will be impossible
anyway).

To upload data, all you need to do is use the `sore` command-line utility:
```bash
sore -i local_file.fits -o testing/2026-01-01/local_file.fits
```
The `-i` input file is the path on your local filesystem. The `-o` output
filename is the position in the global namespace that your file will take.
Your file will then be uploaded, over the internet, to NERSC, where it will
appear at `sore/$GITHUB_USERNAME/testing/2026-01-01/local_file.fits`. It will
shortly be copied to other sites.

Developed with the Codex/OpenAI 5.6 generation models.

Getting set up
--------------

To get set up, you will first need to install the `soreplicator` python package:
```bash
uv pip install soreplicator
```
To authenticate with the system, you will need a username and password. These are
_secrets_ and should never be shared with anyone else. To get your username
and password, you will need to log into the web interface at
https://sore.simonsobservatory.org/setup. Here, you will be able to get your username
and password for the `sore` server. Your username will always be the same
as your GitHub username.

`sore` includes a utility to store these secrets for re-use in a safe location.
We recommend using this. To set up `sore` you should call:
```bash
sore --setup
```
This will allow you to paste in your username and password, which will be saved
in your config location (usually `~/.config/sore.json`) with `600` permissions.

Deleting Files
--------------

Sometimes you will want to remove files from the system, because they are old and
no longer used. You can do this with the `sore` command-line tool:
```bash
sore -d testing/2026-01-01/local-file.fits
```
Which will ask you to input `y/n` for whether you are sure you want to delete
this file in the global namespace. Note that you can only change your own files,
not anyone elses. If you are sure and want to avoid the prompt, you can use
```bash
sore -d testing/2026-01-01/local-file.fits -f
```
to force-delete without needing to respond to the prompt.

Administrator Information
-------------------------

Setting up `sore` is easy. We provide a containerized version of the server. The most
important thing to realize is that we provide one bucket per user, with the bucket
name the same as their GitHub username. This allows us to set bucket-level permissions
for each user, allowing them only `PUT`, `GET`, and `DELETE` access to these buckets,
with `GET` permissions for all buckets.

Included with this repository is a `nginx` configuration file that allows you to mount:
- Versity's S3 Gateway on /
- The provided `sore` server on `/setup`.
- Sharing of directories for `.well-known` challenges.

We authenticate users using the Simons Observatory authentication framework `soauth`.

To set up `sore`, you will need an administrator credential (ideally not the root
credential) that allows the creation of buckets and users with various levels of
permissions. You should set the following environment variables:

- `SORE_GATEWAY_USERNAME`: the username of the account that can create other accounts
- `SORE_GATEWAY_PASSWORD`: the password of the account that can create other accounts
- `SORE_GATEWAY_REMOTE_LOCATION`: the location of the gateway on the open internet
- `SORE_GATEWAY_LOCAL_LOCATION`: the location of the gateway on the local network
- `SORE_ADMIN_GRANT`: the `soauth` grant for administrator priviliges (`sore:admin`)
- `SORE_USER_GRANT`: the `soauth` grant required for users to create accounts (`sore:user`)

We will only communicate with the gateway on the local network and never over the
open internet. The `REMOTE_LOCATION` is just for providing information in the UI.

### Administrator Panels

Within the Web UI, you can see information on all of the user accounts that have
been created to date, and the buckets along with their sizes. This information is
pulled dynamically from the S3 gateway and is not stored separately for the `sore`
server so the `sore` server is effectively stateless. You will require the
`SORE_ADMIN_GRANT` to view this page.

Implementation
--------------

This repository contains the complete gateway edge, account service, and client:

- `docker-compose.yml` runs VersityGW with a POSIX directory, sidecar metadata,
  IAM, its WebGUI, the SORE server, and nginx.
- `soreplicator/server/` is a stateless FastAPI service. It authenticates with `soauth`, creates
  or rotates a VersityGW account, creates the matching bucket, and maintains a
  bucket policy that gives every provisioned account read access while preserving
  owner write/delete access.
- `soreplicator/client/` supplies the `sore` command. Large files use four concurrent S3
  multipart transfers with a terminal progress bar; smaller files use a regular
  PUT. Both paths write conditionally, so an existing object cannot be
  overwritten. Configuration is written atomically with mode `600` and insecure
  remote HTTP endpoints are rejected.
- The `/setup/admin` view queries users, buckets, object counts, and sizes directly
  from VersityGW. No credentials or inventory are stored in an application database.

The downstream Globus replication job is intentionally outside this repository's
scope; it can consume the gateway's shared POSIX directory independently.

### Local stack

The included compose topology is a loopback-only development environment:

```bash
cp .env.example .env
# Change the secrets and SORE_DEVELOPMENT_USERNAME in .env first.
docker compose up -d --build
```

Open the following local endpoints:

- SORE setup: `http://sore.localhost:8080/setup/`
- VersityGW WebGUI through nginx: `http://gateway.localhost:8080/`
- S3 through nginx: `http://s3.localhost:8080/`

The WebGUI is preconfigured to use `s3.localhost` for S3 and `admin.localhost`
for its admin API. Log in with the gateway administrator values from `.env`.
Persistent objects, versions, IAM data, and metadata are stored under `./data`
by default. Set `SORE_DATA_DIRECTORY` to share another host directory.

The compose stack deliberately uses the fixed development identity named by
`SORE_DEVELOPMENT_USERNAME`; nginx publishes only on `127.0.0.1`. Development
authentication cannot start unless `SORE_DEBUG=true`. Never expose this mode to a
network.

### Production server configuration

The Docker image runs as an unprivileged user with a read-only-compatible root
filesystem. For production, set `SORE_AUTH_MODE=soauth`, `SORE_DEBUG=false`, and
provide these settings to the server container:

| Setting | Purpose |
| --- | --- |
| `SORE_GATEWAY_USERNAME` / `SORE_GATEWAY_PASSWORD` | Versity administrator credential |
| `SORE_GATEWAY_REMOTE_LOCATION` | Public S3 URL displayed to clients |
| `SORE_GATEWAY_LOCAL_LOCATION` | Private S3 URL used by the server |
| `SORE_GATEWAY_ADMIN_LOCATION` | Private Versity admin API URL |
| `SORE_GATEWAY_REGION` | SigV4 region; defaults to `us-east-1` |
| `SORE_APP_BASE_URL` | Public application URL, including `/setup` |
| `SORE_AUTHENTICATION_BASE_URL` | SOAuth service URL |
| `SORE_APP_ID` / `SORE_CLIENT_SECRET` | SOAuth application credential |
| `SORE_PUBLIC_KEY_FILE` | Mounted SOAuth public key path |
| `SORE_KEY_PAIR_TYPE` | SOAuth key type; defaults to `Ed25519` |
| `SORE_ADMIN_GRANT` / `SORE_USER_GRANT` | Required grants |

Terminate TLS at nginx or an upstream load balancer, keep the admin API private,
replace every development secret, and adapt `nginx/nginx.conf` to the production
hostnames. The `.well-known` directory is mounted read-only for ACME challenges.

### Development and verification

```bash
uv sync --extra dev
uv run pytest
uv run ruff check .
npm install
npm run build:css
docker compose config
```

Tailwind output and htmx are vendored under `soreplicator/server/static`, so the setup UI does
not depend on third-party CDNs. The Tailwind palette is the colorblind-friendly
SO palette from `socolors`.

### Releases

GitHub Actions runs Ruff and the test suite on Python 3.11–3.13 for pull requests
and pushes to `main`, and verifies that the committed Tailwind stylesheet is current.
To publish a release, update the version in `pyproject.toml`, merge it to `main`,
and publish a GitHub Release.

The publishing workflow uses [PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/)
instead of a stored API token. Before the first release, configure a GitHub Actions
Trusted Publisher for this repository and the `pypi` environment in the
`soreplicator` PyPI project.
