Metadata-Version: 2.4
Name: user-data-ingest-cli
Version: 0.1.84
Summary: CLI tool for ingesting user data via the API
Author: ASTRON SDC
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: requests
Requires-Dist: fsspec
Requires-Dist: PySide6
Dynamic: license-file

# User Data Ingest CLI (udicli)

Command-line client for uploading data into LOFAR 2.0's Long-Term Archive. Built for projects with non-standard pipelines or independent processing resources that need to submit data while adhering to LTA schemas.

See the [backend documentation](https://git.astron.nl/astron-sdc/user-ingest-backend) for more context. There's also a [web interface](https://sdchealth.fuse-astron.src.surf-hosted.nl/user-data-ingest/) if you prefer that.

## Getting Started

Python 3.13+ required ([download here](https://www.python.org)).

```bash
pip install user-data-ingest-cli
udicli login
```

`udicli` is now on your PATH.

## Commands

```bash
udicli login          Authenticate with your SRAM application token
udicli whoami         Show your account details
udicli status         Check server connectivity
udicli list-projects  View projects you are a member of
udicli view-requests  View a summary of ingest requests
udicli connect        Configure access to your data storage
udicli ingest         Create and submit an ingest request
udicli logout         Remove cached credentials
```

For help on any command:

```bash
udicli <command> --help
```

Use `-v` flag with any command to see raw requests and responses:

```bash
udicli -v status
```

## Authentication

Signing in to User Data Ingest with the `login` command  will prompt you to login with the account from an institution you are affiliated with.
If you do not have an account with an institution that is part of the EduGAIN federation, you can create an account with one of the eduID services (e.g. the Dutch one).

To request access to this service, you can use this link : https://sram.surf.nl/registration?collaboration=e80b9e70-62f8-4b05-890e-c40dd9d2ac68

After the membership is approved you can go to the [Application token page](https://sram.surf.nl/collaborations/7733/tokens) and click " Create application token". Make sure to copy the given token somewhere safe as this will not be visible
a second time. To then log in via the cli tool you can either put it in a `.ingestingrc` file in your home directory or give it as an argument to the login command, as explained below.
![Create an application token page](docs/images/create-token.png)

`login` looks for your SRAM application token in this order:

1. `--token` (optional) flag: `udicli login --token <your-token>`

2. `.ingestingrc` file in your home directory with `api_token=<your-token>`
   macOS/Linux: `~/.ingestingrc`
   Windows: `C:\Users\<username>\.ingestingrc`
   usage: `udicli login`

3. Interactive prompt: provides a link to SRAM and asks you to paste the token

Once verified, the token is cached locally at `~/.config/user-ingest-cli/credentials-<env>`, so you won't need to authenticate again.

Switching environments with `--env dev` or `--env prod` keeps sessions separate.
eg usage : `udicli --env dev login`

## Ingesting Data

An ingest request has this structure: project → data products → files. Each file needs a checksum value and type.

### Interactive Mode

Start with no arguments to be guided through the whole process:

```bash
udicli ingest
```

You'll select a project, site, checksum type, and build data products interactively.

## Note:
You should provide (absolute or relative) file locations.
For all the ways mentioned below you can use glob patterns like "data/*.m5", which selects all m5 files from folder data.
For now the tool checks for * or ? in the path. It does not check [..] and does not support recursion.

### Command-Line Flags

Provide all details via flags .

```bash
udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --file-format MEASUREMENT_SET --product-type VisibilityDataProduct  --files test_files/file1.m5 --files test_files/file2.m5
udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --file-format MEASUREMENT_SET --product-type VisibilityDataProduct  --files test_files/file1.m5 --files test_files/file2.m5
udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --file-format MEASUREMENT_SET --product-type VisibilityDataProduct  --files test_files/file1.m5 --files /home/<user>/Downloads/file2.m5
udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --file-format MEASUREMENT_SET --product-type VisibilityDataProduct  --files "test_files/*.m5"
```

Repeat `--files` for multiple files. Don't repeat other flags (they won't stack; only the last one counts).

### Piping File Lists

Use `--files-from -` to read from stdin:

```bash
find test_files -name "*.m5" -mtime -1 | udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --product-type VisibilityDataProduct --file-format MEASUREMENT_SET --files-from -
find test_files -name "*.m5"| udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --product-type VisibilityDataProduct --file-format MEASUREMENT_SET --files-from -
```

### CSV File

Pass a file with one path per line:

```bash
udicli ingest --project APPPP_001 --site surf --checksum-type MD5 --product-type VisibilityDataProduct --file-format MEASUREMENT_SET --files-from test_config_files/config3.csv
```

Empty lines in the file are ignored.

### JSON Config File

For complex scenarios with multiple data products and filters:

```json
{
  "project": "APPPP_001",
  "site": "surf",
  "checksum_type": "MD5",
  "data_products": [
    {
      "product_type": "VisibilityDataProduct",
      "format": "MEASUREMENT_SET",
      "files": [
        "test_files/file*.m5"
      ]
    },
    {
      "product_type": "VisibilityDataProduct",
      "format": "MS",
      "files": [
        "test_files/*.MS"
      ],
      "filters": {
        "min_size": "1000000",
        "max_size": 100000000,
        "modified_after": "2024-01-01",
        "exclude": [
          "*test*.m5",
          "*backup*.m5"
        ]
      }
    }
  ]
}
```

Run with:

```bash
udicli ingest --inputfile test_config_files/inputFile1.json
udicli ingest --inputfile test_config_files/inputFile2.json
```

## Filters (JSON Config Only)

Filters refine file selection after glob patterns expand. All are optional; a file must pass every filter to be included.

```json
"filters": {
  "min_size": 1000000,
  "max_size": 100000000,
  "modified_after": "2026-07-22",
  "include": ["file1.m5", "file2.m5"],
  "exclude": ["*test*.m5", "*backup*.m5"]
}
```

`min_size` and `max_size` accept numbers or strings (e.g., `1000000` or `"1000000"`).

`modified_after` takes ISO 8601 format: `"2026-07-22"` or `"2026-07-22T14:00:00"`.

`include` and `exclude` use glob patterns (`*.m5`, `file?.m5`). Character classes `[..]` and recursive globs `**` are not supported.

## Request Body Format

What gets sent to the server (for reference):

```json
{
  "project_id": "DEV__P_C1_003",
  "site": "surf",
  "checksum_type": "MD5",
  "estimated_total_size_bytes": 1536770867,
  "data_products": [
    {
      "data_product_type": "VisibilityDataProduct",
      "files_format": "MEASUREMENT_SET",
      "files": [
        {"filename": "obs1.ms", "checksum_value": "abc123..."},
        {"filename": "obs2.ms", "checksum_value": "def456..."}
      ]
    }
  ]
}
```


With `--noinput`, the tool fails fast if required flags are missing. All validation errors are reported before any files are sent.

Piped input (e.g., `find ... | udicli ingest ... --files-from -`) automatically enables non-interactive mode.

## Testing and Feedback

We're actively developing this tool. As you test, please share your thoughts on these questions:

1. What do your processing resources typically include and how do you connect to them?

2. In a typical cycle, how many files and data products would you upload?

3. Should duplicate files be automatically filtered out?

4. What's the usual folder and file structure of pipeline output? This helps us improve the ingest interface.

5. Would concurrent request management be useful? For example, draft one request, switch to another, then submit both when ready.

6. File format validation isn't implemented yet. We're waiting on the LOFAR Data Working Group to approve the list of valid data products and formats.

7. Should a data product with zero files be rejected with an error or allowed with a warning?

8. What validation rules matter for data you provide via flags or input files? Check numerical values, empty values, consistency between flags?

9. If a file path fails during processing (permissions, disappeared since resolving), should it be skipped with a warning or should the whole request fail?

## Development

Clone and set up:

```bash
git clone <this-repo>
cd user-data-ingest-cli
python -m venv .venv
source .venv/bin/activate
make bootstrap
make init
pip install -e .
```

Runtime dependencies live in `pyproject.toml`'s `dependencies`. `requirements/base.txt` is just a pinned lock file compiled from it, used to keep dev/CI environments reproducible;
`requirements/dev.txt` adds dev-only tooling (pytest, ruff, black, mypy, pre-commit) on top and is compiled from `requirements/dev.in`.

After changing dependencies, recompile the lock files and reinstall:

```bash
make compile-requirements
pip install -r requirements/dev.txt
pip install -e .
```

Run linting, formatting, type-checking, and tests:

```bash
ruff check src tests
black src tests
mypy src
pytest
```

### Local State

The SRAM token obtained via `login` is written to `~/.config/user-ingest-cli/credentials-{env}` (a separate file per `--env`) and stays there until you run `udicli logout` or delete the file yourself; there's no client side expiry check, so a revoked or expired token will just fail on the next request to the API.
Project codes are never cached: `list-projects` and the interactive project prompt in `ingest` both call the API every time, so they always reflect your current memberships.
Everything else, such as which environment you're targeting or which API base URL to call, lives in the in memory `Config` object, rebuilt from `constants.py` and CLI flags/environment variables on every invocation.

## Deployment

Every pipeline builds the package (`package_files`) with an auto-incrementing version (`0.1.<commit count>` — no manual version action needed). From there, two manual jobs are available in the pipeline's `publish` stage:

**Test deploy to GitLab** — click `publish_on_gitlab` to start the pipeline. Uploads to this project's own Package Registry, authenticated automatically via `CI_JOB_TOKEN`

```bash
pip install user-data-ingest-cli --index-url https://__token__:<your-personal-access-token>@git.astron.nl/api/v4/projects/1019/packages/pypi/simple
```

**Release to PyPI** — push a tag, then click `publish_on_pypi`:

```bash
git tag v0.2.0
git push origin v0.2.0
```

That job only appears on tag pipelines and uploads to the real [pypi.org](https://pypi.org/project/udicli/) using a `PYPI_TOKEN` CI/CD variable (set under Settings → CI/CD → Variables, masked + protected). Once it's up:

```bash
pip install user-data-ingest-cli
```
