Metadata-Version: 2.4
Name: usergenerated
Version: 2.1.4
Summary: Utilities to prepare User Generated collections for DestinE Data Lake STAC workflows.
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: pystac[orjson,validation]
Requires-Dist: shapely
Requires-Dist: requests
Requires-Dist: beautifulsoup4
Requires-Dist: netCDF4
Requires-Dist: boto3
Requires-Dist: botocore
Requires-Dist: python-dotenv
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pip-tools; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Provides-Extra: collection-tests
Requires-Dist: tqdm; extra == "collection-tests"
Requires-Dist: destinelab; extra == "collection-tests"
Provides-Extra: deployed
Requires-Dist: destinelab; extra == "deployed"
Provides-Extra: magic
Requires-Dist: python-magic; extra == "magic"

# User Generated Data Process

Table of Contents:
1. [Overview](#overview)
2. [User Workflow](#user-workflow)
3. [Required Structure](#required-structure)
4. [Metadata Files](#metadata-files)
   - [collection.json](#collectionjson)
   - [collection_config.json](#collection_configjson)
5. [Generate Item Metadata](#generate-item-metadata)
6. [Regenerate Item Metadata in S3](#regenerate-item-metadata-in-s3)
7. [Maintainer Notes](#maintainer-notes)
8. [Optional S3 Upload](#optional-s3-upload)
9. [Runtime Initialization](#runtime-initialization)
10. [Troubleshooting](#troubleshooting)
11. [External References](#external-references)



This project helps you prepare a User Generated collection for DestinE Data Lake review.

Use this guide to:
1. Structure your collection folders and data assets.
2. Configure metadata files.
3. Generate STAC Item metadata.

For contributor and maintenance documentation, see [docs/development.md](docs/development.md).

For a feature overview (data provider and operations team capabilities), see [docs/USERGENERATED_FEATURES.md](docs/USERGENERATED_FEATURES.md).

For an overview of how the package is structured and how it interacts with S3 and the deployed STAC API,
see [docs/USERGENERATED_HLA.md](docs/USERGENERATED_HLA.md).

## User Workflow

1. Create your collection root folder using the provided collection id, for example `EO.XXX.YYY.ZZZ`.
2. If needed, place the collection in a subfolder under your current working directory and pass that subfolder with `--collection-path`.
3. Add metadata files under `metadata/`.
4. Organize data assets under `data/` according to your configured folder level.
5. Run [generate_item_metadata.py](generate_item_metadata.py) to validate and generate STAC Items under `metadata/items/`.

If you mirror a collection from S3 before regenerating items, mirrored data folders may contain `usergenerated_s3_folder_manifest.jsonl` files instead of placeholder dummy assets. Item generation detects these manifests automatically and uses their entries as the authoritative asset list.

Reference collection structure is available in [EO.XXX.YYY.ZZZ](EO.XXX.YYY.ZZZ).

## Required Structure

Expected top-level structure:

```text
EO.XXX.YYY.ZZZ/
  data/
  metadata/
    collection.json
    collection_config.json
    items/
```

Notes:
1. Collection ids are case-sensitive and typically uppercase with `.` separators.
2. The `metadata/items/` folder is where generated Item JSON files are written.

## Metadata Files

### collection.json

#### How to Complete `collection.json`

Use `metadata/collection.json` as the authoritative STAC Collection metadata file for your dataset.

---

#### Required STAC Collection fields

The following fields **must** be completed and valid:

1. `type` — Must be set to `Collection`.
2. `stac_version` — Use the current version (`1.1.0`). `generate-item-metadata` rejects the
   collection if `stac_version` is missing or set to anything else.
3. `id` — Your collection identifier. This must match:
   - your root folder name  
   - `collection_config.json > id`
4. `title` — A human-readable name for your collection.
5. `description` — A detailed description of the dataset.
6. `dedl:short_description` — A concise version of the description. Displayed in the DestinE DataLake UI.
7. `license` — The license under which the data is distributed.
8. `extent` — Must include both spatial and temporal components.
9. `links` — Must be present as an array.
10. `assets` — Include if you have collection-level assets; otherwise use `{}`.

---

#### Recommended best practices

- Keep the `id` consistent across:
  - the collection root folder name  
  - `collection_config.json > id`
- Use a clear `title` and a meaningful `description` that explains:
  - dataset content  
  - data source(s)  
  - intended use cases
- Ensure `extent` is accurate so item-level metadata is interpreted correctly.
- Only update necessary `links` (typically `describedby` and `cite-as`).
- Keep `type` and `stac_version` unchanged unless upgrading STAC versions.

---

#### Field-specific guidance

##### `links`
- Always include at least:
  - `license` → link to license terms
  - `cite-as` → DOI or citation reference
  - `describedby` → external dataset documentation
- Ensure links are stable and publicly accessible.
- Use meaningful `title` values so users understand what each link represents.

---

##### `extent`
- **Spatial (`bbox`)**:
  - Format: `[west, south, east, north]`
  - Use WGS84 coordinates (EPSG:4326).
  - Ensure the bounding box fully covers your dataset.

- **Temporal (`interval`)**:
  - Use ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`).
  - Use:
    - a closed interval `[start, end]` for fixed datasets
    - `[start, null]` for ongoing datasets

---

##### `license`
- Prefer SPDX identifiers (e.g. `CC-BY-4.0`, `MIT`, `Apache-2.0`).
- Use:
  - `proprietary` for restricted datasets
  - `various` if multiple licenses apply
- If using `proprietary` or `various`:
  - explain clearly in `description`
  - provide license links in `links`

---

##### `providers`
- List all organizations involved in the dataset lifecycle.
- Common roles:
  - `producer` → created the data
  - `processor` → processed or transformed the data
  - `licensor` → owns or distributes the data rights
  - `host` → (optional) hosting platform
- Provide:
  - `name` (required)
  - `url` (recommended)
- Include multiple providers if applicable rather than combining roles into one.

---

##### `sci:publications`
- Use this field to reference scientific publications or datasets.
- Include:
  - `sci:doi` → DOI identifier
  - `sci:citation` → human-readable citation
- Add multiple entries if there are several related publications.
- Ensure DOIs resolve correctly (test the links).

---

##### `keywords`
- Provide relevant, searchable terms describing the dataset.
- Use:
  - domain-specific terminology (e.g. *Land Cover*, *Atmospheric Data*)
  - common synonyms where helpful
- Avoid overly generic terms (e.g. “data”, “analysis”).

---

##### `assets`
- Use for collection-level resources (not item-level data).
- Common examples:
  - `thumbnail` → preview image
  - `overview` → documentation PDF
- Include:
  - `href` (URL)
  - `type` (MIME type, e.g. `image/png`)
  - `roles` (e.g. `thumbnail`, `overview`)
- If no assets are needed, use an empty object:
  ```json
  "assets": {}
  ```

### collection_config.json

This controls how Item metadata generation behaves. Base example in [EO.XXX.YYY.ZZZ/metadata/collection_config.json](EO.XXX.YYY.ZZZ/metadata/collection_config.json).

## Configuration Options

The following options are user-facing and supported during item generation:

| Key | Type | Default | Description |
|---|---|---|---|
| `id` | string | required | Collection id. Must match your collection root folder id. |
| `item_folder_level` | string | `DD` | Where item folders are discovered. Allowed values: `YYYY`, `MM`, `DD`, `NONE`. |
| `item_config_optional` | boolean | `false` | If `false`, each item folder is expected to contain `item_config.json`. |
| `item_asset_ignore_list` | list[string] | `[]` | File names to ignore when creating STAC assets. |
| `media_type_overrides` | object | `{}` | Optional extension to MIME type map used before default detection, e.g. `{ ".grib2": "application/x-grib2" }`. |
| `role_overrides` | object | `{}` | Optional MIME type to role map. Overrides default `data`/`metadata` classification. Allowed roles: `data`, `metadata`, `thumbnail`, `overview`. E.g. `{ "application/x-grib2": "data" }`. File-name regex patterns (`thumbnail_regex`, `overview_regex`) still take priority. |
| `thumbnail_regex` | string | unset | Regex used to detect thumbnail assets and assign thumbnail role. |
| `overview_regex` | string | unset | Regex used to detect overview assets and assign overview role. |
| `additional_property_keys` | list[string] | `[]` | Additional property suffix keys parsed from item folder names (standard mode). |
| `item_folder_naming_convention_type` | string | `standard` | Naming convention mode. Allowed values: `standard`, `non_standard_1`. |
| `properties` | object | `{}` | Additional properties merged into each generated Item. |
| `bbox` | list[number] | unset | Bounding box `[minx, miny, maxx, maxy]` applied to generated items. |
| `item_date_overide` | string | unset | Simplified mode override date in `YYYYMMDD` format. |
| `asset_href_prefix` | string | unset | Optional prefix prepended to every generated asset href, e.g. a public bucket URL such as `https://s3.central.data.destination-earth.eu/`. A trailing slash is normalized; an empty string behaves as unset. Can be overridden at run time — see [Overriding the prefix at run time](#overriding-the-prefix-at-run-time). |

Important: the accepted key name is `item_date_overide` (spelling as shown).

## Folder Granularity Options

Set `item_folder_level` in `collection_config.json`.

1. `DD`: item folders are expected under `data/YYYY/MM/DD/<item_folder>`.
2. `MM`: item folders are expected under `data/YYYY/MM/<item_folder>`.
3. `YYYY`: item folders are expected under `data/YYYY/<item_folder>`.
4. `NONE`: simplified mode. Any folder directly under `data/` becomes an Item.

## Structure Conformance Check

Before any Item is written, `generate-item-metadata` checks every file under `data/` against the
configured `item_folder_level` and fails if anything does not fit. It also detects Item folders that
contain no files at all — an empty folder would otherwise silently produce a STAC Item with zero
assets. All problems are reported in a single run, so a mis-structured collection can be fixed in one
pass rather than one error at a time.

Nothing is written to `metadata/items/` when the check fails.

| Reported reason | What it means | How to fix it |
|---|---|---|
| `empty_item_folder` | An Item folder contains no files (not even in nested subfolders). | Remove the folder, or add the Item's asset files to it. |
| `loose_file` | A file sits above item level, e.g. directly in `data/2024/11/15/`. It belongs to no Item, so it would never become an asset. | Move it into an item folder, or remove it. |
| `invalid_year_folder` | A folder under `data/` is not a 4-digit year. | Rename it, e.g. `2024`. |
| `invalid_month_folder` | A month folder is not a zero-padded `01`-`12`. | Rename it, e.g. `01`, not `1`. |
| `invalid_day_folder` | A day folder is not a zero-padded `01`-`31`. | Rename it, e.g. `05`, not `5`. |
| `invalid_calendar_date` | The `YYYY/MM/DD` folders are well-formed but not a real date, e.g. `2024/02/30`. | Correct the date folders. |
| `invalid_item_folder_name` | The item folder name cannot be parsed as an item id: it does not start with the collection id, its datetime segment is malformed, or its `__` suffixes do not match `additional_property_keys`. | Rename it to `<collection_id>_<start_datetime>[_<end_datetime>]`, see Standard Mode Example. |

A folder is reported once however many files sit under it, and reporting stops after 50 findings.

The same check runs during `regenerate-item-metadata-s3`, against the local mirror of the bucket. Mirror
bookkeeping files (`usergenerated_s3_folder_manifest.jsonl`) are not collection content and are never
reported: the objects they describe are checked under their real names, so a reported path is always one
that exists in your bucket. An empty Item folder in S3 is reported too: a folder created in the S3 console
(or by tools such as `s3cmd`) is a zero-byte marker object whose key ends in `/`, and the mirror recreates it
as an empty local folder. A marker on a folder that also holds objects is not reported.

When `item_folder_level` is `NONE`, simplified mode imposes no layout, so every folder directly under
`data/` is treated as an Item; the layout checks are skipped, but empty Item folders are still reported.

## Standard Mode Example

```json
{
  "id": "EO.XXX.YYY.ZZZ",
  "item_asset_ignore_list": ["item_config.json"],
  "media_type_overrides": {
    ".grib2": "application/x-grib2"
  },
  "item_config_optional": false,
  "item_folder_level": "DD",
  "thumbnail_regex": "^thumbnail",
  "overview_regex": "^overview",
  "additional_property_keys": []
}
```

## Simplified Mode Example

Simplified mode should only be used with agreement from support.

```json
{
  "id": "EO.XXX.YYY.ZZZ",
  "item_asset_ignore_list": ["item_config.json"],
  "item_folder_level": "NONE",
  "item_date_overide": "20240101",
  "thumbnail_regex": "^thumbnail",
  "overview_regex": "^overview",
  "additional_property_keys": [],
  "bbox": [-10.0, 35.0, 10.0, 60.0]
}
```

Simplified mode datetime behavior:
1. If `item_date_overide` is set, that date is used for all generated Items.
2. Otherwise, if the first folder level under `data/` is a year, `YYYY-01-01` is used.
3. Otherwise, the generation date is used.

## Asset Href Prefix

By default, generated asset hrefs are relative to the collection root, e.g.:

```json
"href": "data/2024/11/15/EO.XXX.YYY.ZZZ_20241115T000000_20241115T235959/20241115.png"
```

Set `asset_href_prefix` in `collection_config.json` to prepend a fixed prefix (typically a public bucket
URL) to every generated asset href:

```json
{
  "id": "EO.XXX.YYY.ZZZ",
  "asset_href_prefix": "https://s3.central.data.destination-earth.eu/"
}
```

which produces:

```json
"href": "https://s3.central.data.destination-earth.eu/data/2024/11/15/EO.XXX.YYY.ZZZ_20241115T000000_20241115T235959/20241115.png"
```

The prefix applies to every generated asset regardless of role (`data`, `metadata`, `thumbnail`,
`overview`); item/collection `self` links are unaffected. This works identically for
`generate-item-metadata`, `regenerate-item-metadata-s3`, and `promote_collection` — whichever
`collection_config.json` is in effect for the collection being processed is the one consulted.

### Overriding the prefix at run time

Every entry point also accepts an explicit prefix that **overrides** any `asset_href_prefix` set in
`collection_config.json` *or* in a per-item `item_config.json`. No config file is modified — in S3 the
source `collection_config.json` stays as it is; only the generated item hrefs change.

| Entry point | Override |
|---|---|
| `generate-item-metadata` | `--asset-href-prefix URL` |
| `ItemGenerator(...)` | `asset_href_prefix_override="URL"` |
| `regenerate-item-metadata-s3` | `--asset-href-prefix URL` |
| `ItemGeneratorS3(...)` | `asset_href_prefix_override="URL"` |
| `promote_collection(...)` | `asset_href_prefix="URL"` |
| `script_s3_promote_collection.py` / `script_s3_regenerate_item_metadata.py` | `S3_ASSET_HREF_PREFIX` env var |

- Omitted / `None` (or the env var unset): no override, the configured value is used as before.
- An empty string (`--asset-href-prefix ""`, `S3_ASSET_HREF_PREFIX=`): forces **no** prefix, even
  when one is configured.
- The same trailing-slash normalization applies.

```python
promote_collection(
    collection_id="EO.XXX.YYY.ZZZ",
    ...,
    on_ready=on_ready,
    asset_href_prefix="https://s3.central.data.destination-earth.eu/my-target-bucket/EO.XXX.YYY.ZZZ/",
)
```

## Per-Item Configuration

Each item folder can include an `item_config.json` file. Typical per-item overrides:
1. `bbox`
2. `properties`
3. Asset-related regex fields when needed

When present, item-level values override collection-level values.

## Generate Item Metadata

### CLI Usage

Primary command (after installing the package):

```bash
generate-item-metadata EO.XXX.YYY.ZZZ
```

This default expects the collection at `./EO.XXX.YYY.ZZZ`.

Optional collection subfolder path:

```bash
generate-item-metadata EO.XXX.YYY.ZZZ --collection-path toto
```

With this option, the collection is resolved at `./toto/EO.XXX.YYY.ZZZ`.
For safety, `--collection-path` is currently restricted to subfolders under the current working directory.

Optional bucket override:

```bash
generate-item-metadata EO.XXX.YYY.ZZZ --bucket-name usergenerated-proposal-eo.xxx.yyy.zzz
```

Optional asset href prefix override (outranks any configured `asset_href_prefix`; `""` forces no prefix —
see [Overriding the prefix at run time](#overriding-the-prefix-at-run-time)):

```bash
generate-item-metadata EO.XXX.YYY.ZZZ --asset-href-prefix "https://s3.central.data.destination-earth.eu/my-bucket/EO.XXX.YYY.ZZZ/"
```

Compatibility command (run from the project root):

```bash
python generate_item_metadata.py EO.XXX.YYY.ZZZ
```

Compatibility command with collection subfolder:

```bash
python generate_item_metadata.py EO.XXX.YYY.ZZZ --collection-path toto
```

### Programmatic Usage

You can also use `ItemGenerator` directly in your Python scripts:

```python
from usergenerated import ItemGenerator

# Create a generator for your collection
generator = ItemGenerator("EO.XXX.YYY.ZZZ")

# Generate Item metadata
generator.run()

# Resolve collection from ./toto/EO.XXX.YYY.ZZZ
generator_with_path = ItemGenerator("EO.XXX.YYY.ZZZ", collection_path="toto")
generator_with_path.run()

# With optional bucket name override
generator_with_bucket = ItemGenerator("EO.XXX.YYY.ZZZ", override_bucket_name="custom-bucket")
generator_with_bucket.run()

# Also write a reordered/normalised copy to output/collection_reordered.json (off by default)
generator_reordered = ItemGenerator("EO.XXX.YYY.ZZZ", save_reordered_collection=True)
generator_reordered.run()
```

Generated output is written to:

```text
<COLLECTION_ID>/metadata/items/*.json
```

The `ItemGenerator` class is the primary API for programmatic use. The CLI command is a thin wrapper around this class.

## Regenerate Item Metadata in S3

Use `ItemGeneratorS3` to regenerate STAC Item metadata from bucket data while restricting updates to Item metadata only. This is useful when you need to update item generation logic or fix generated metadata without modifying data files in S3.

### CLI Usage

Primary command (regenerate items in S3 for your collection):

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ
```

This command:
1. Mirrors collection data from S3 to a local folder.
2. Regenerates STAC Item metadata locally with your configured settings — the result is exactly the items backed
   by a folder under `data/`, so an item whose data folder has been removed or renamed is dropped.
3. Backs up existing S3 items by default.
4. Replaces only the `metadata/items` prefix in S3 (never modifies data files).

Optional dry-run (validate without modifying S3):

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run
```

Optional bucket name override:

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --bucket-name custom-bucket-name
```

Optional local mirror folder (default: `tmp_structure_mirror`):

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --local-root "./my_mirror"
```

Disable backup:

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-backup-items
```

Skip the `ready_for_transfer` safety check (see [Safety Behavior](#safety-behavior); enabled by default):

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-use-ready-for-transfer-flag
```

Override the asset href prefix for the regenerated items (the `collection_config.json` in S3 is not modified;
`""` forces no prefix):

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --asset-href-prefix "https://s3.central.data.destination-earth.eu/my-bucket/EO.XXX.YYY.ZZZ/"
```

Demonstration script usage (env-first configuration via `.env`/environment variables, see
`script_s3_regenerate_item_metadata.py` for the full list of `S3_*` variables):

```bash
python script_s3_regenerate_item_metadata.py
```

This script is intentionally a thin example of programmatic `ItemGeneratorS3` usage.
For routine operations, prefer the `regenerate-item-metadata-s3` console command.

The demo script additionally supports `USE_READY_FOR_TRANSFER_FLAG` (or the `use_ready_for_transfer_flag`
keyword argument to `run_regenerate_item_metadata`) to toggle the `ready_for_transfer` safety check described
below — see [Safety Behavior](#safety-behavior). The primary console command exposes the same toggle via
`--use-ready-for-transfer-flag`/`--no-use-ready-for-transfer-flag` (shown above).
Both also accept an asset href prefix override (`S3_ASSET_HREF_PREFIX` / `--asset-href-prefix`) — see
[Overriding the prefix at run time](#overriding-the-prefix-at-run-time).

### Programmatic Usage

You can also use `ItemGeneratorS3` directly in your Python scripts:

```python
from usergenerated import ItemGeneratorS3

# Create a generator for your collection
generator = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    dry_run=True,  # Validate without modifying S3
)

# Regenerate Item metadata
generator.run()

# With explicit bucket name
generator_with_bucket = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    bucket_name="custom-bucket",
    backup_items=True,
)
generator_with_bucket.run()

# Skip the ready_for_transfer safety check (see Safety Behavior below); enabled by default
generator_without_gate = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    use_ready_for_transfer_flag=False,
)
generator_without_gate.run()
```

### Safety Behavior

The regeneration workflow includes several safety measures:

1. Validates credentials and bucket structure before processing.
2. Mirrors S3 content locally and ensures local processing under `<collection_id>/`.
3. By default (`use_ready_for_transfer_flag=True`), requires `"ready_for_transfer": true` in the mirrored
   collection's `metadata/collection_config.json`; aborts with an error if the field is missing or `false` (this
   check applies even with `--dry-run`). Skip this check via `--no-use-ready-for-transfer-flag` on the CLI, the
   `use_ready_for_transfer_flag=False` constructor argument, or `USE_READY_FOR_TRANSFER_FLAG=false` for the demo
   script.
4. Clears the mirrored copy of `metadata/items`, then regenerates Item metadata locally with existing generation
   logic. This is what makes regeneration a replacement rather than a merge: an item in S3 with no matching item
   folder under `data/` is reported (`... have no matching item folder under 'data/' ...`) and not re-uploaded,
   instead of surviving every run. The S3 backup in step 5 is the recovery path if a folder was removed by mistake.
5. Backs up existing `<collection_id>/metadata/items` by default (can be disabled).
6. Replaces only `<collection_id>/metadata/items` in S3 (never modifies `data/`).
7. With `--dry-run`, performs local validation and regeneration, then prints planned S3 actions without modifying the bucket.

### Mirroring Behavior

The mirroring process downloads real files selectively to improve performance:

1. Downloads real files only for `metadata/*` and `item_config.json` keys.
2. Writes per-folder manifest files (`usergenerated_s3_folder_manifest.jsonl`) for discovered objects.
3. Uses manifest entries during item generation so full non-metadata assets are not required locally.
4. Supports optional placeholder file creation for non-fetched objects (disabled in the S3 regeneration workflow).
5. Recreates folder-marker objects (keys ending in `/`) as empty local folders, with no manifest, so an empty
   Item folder in the bucket is caught by the structure check.
6. Logs a summary with discovered, downloaded, and failed object counts, plus periodic progress logs during large scans.

## Maintainer Notes

Recent internal refactoring improved testability and packaging workflows.

1. The primary command is now `generate-item-metadata <COLLECTION_ID>`.
2. [generate_item_metadata.py](generate_item_metadata.py) is retained as a compatibility wrapper.
3. Internal logic was modularized into reusable helpers under `usergenerated/` for:
  - collection path resolution
  - item folder discovery by configured granularity
  - item config precedence/loading
  - simplified and hierarchy-based datetime resolution
4. This split enables unit-heavy testing of pure logic while keeping entry-point orchestration thin.

## Optional S3 Upload

Automatic upload is configured at runtime via `ItemGenerator` constructor parameters
or CLI flags (not via module constants).

Programmatic example:

```python
from usergenerated import ItemGenerator

generator = ItemGenerator(
    "EO.XXX.YYY.ZZZ",
    is_upload_s3=True,
    s3_endpoint_url="https://s3.central.data.destination-earth.eu",
    s3_user_generated_bucket_prefix="usergenerated-proposal",
)
generator.run()
```

CLI example:

```bash
generate-item-metadata EO.XXX.YYY.ZZZ \
  --upload-s3 \
  --s3-endpoint-url "https://s3.central.data.destination-earth.eu" \
  --s3-bucket-prefix "usergenerated-proposal"
```

If upload is enabled, create a `.env` file at the project root with credentials:

```bash
AWS_ACCESS_KEY_ID="<your_access_key_id>"
AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"
```

For manual upload workflows, see [README-s3cmd.md](README-s3cmd.md).

S3 utility wrappers are available for programmatic execution around `S3Tools`:

```python
from script_s3_create_bucket import run_create_bucket
from script_s3_move_bucket_contents import run_move_bucket_contents
from script_s3_upload_file_to_bucket import run_upload_file
from script_s3_upload_folder_to_bucket import run_upload_folder

run_create_bucket(
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_upload_file(
  file_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
  object_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_upload_folder(
  folder_path="EO.XXX.YYY.ZZZ/metadata/items",
  target_path="EO.XXX.YYY.ZZZ/metadata/items",
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_move_bucket_contents(
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
  s3_target_prefix="archive",
)
```

### S3 Folder Upload Progress

When using `run_upload_folder(...)` or `S3Tools.upload_folder_to_s3(...)`, uploads now display progress bars:

1. A global `Total files` bar showing overall folder upload progress.
2. Per-file bars (bytes transferred) for active uploads.

Current behavior:

1. Uploads run concurrently with a bounded worker pool.
2. Per-file progress bars are capped to a visible range and positions are reused for large uploads.
3. `usergenerated_s3_folder_manifest.jsonl` files and symlinks are skipped from upload discovery.
4. A final summary logs discovered, uploaded, and failed file counts.
5. Progress bars are drawn only when `stderr` is a terminal. Under a non-interactive runner (CI job, Kubernetes
   pod, cron entry) they are suppressed automatically, so captured logs contain the summary lines rather than
   thousands of bar refreshes. The same applies to the collection copier's `Copying objects` bar.

These wrappers accept explicit parameters first and only fall back to environment variables when a parameter is omitted.

**Regenerate Item Metadata**: Use the CLI command for routine operations (see [Regenerate Item Metadata in S3](#regenerate-item-metadata-in-s3)):

```bash
regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run
```

The `script_s3_regenerate_item_metadata.py` file is kept as a small demonstrator of `ItemGeneratorS3` programmatic usage. Use the CLI command or the `ItemGeneratorS3` class directly for real workflows.

### Copy a Collection Between Buckets

`script_s3_copy_source_bucket_collection_to_target_bucket_collection.py` copies a collection
folder from a source S3 bucket/account to a target S3 bucket/account, using
`usergenerated.s3.s3_collection_copier.CollectionCopier`. It:

- Detects the source bucket's `BucketLayout` (`COLLECTION_ROOT` or `FLAT_ROOT`) and normalizes
  the target to always be `<collection_id>/...` (`COLLECTION_ROOT`), fixing up `FLAT_ROOT`
  sources in the process.
- Never modifies the source bucket.
- **Synchronises** rather than merely appends: an object already in the target is skipped only
  when it is unchanged (same size, and the target copy is not older than the source); a changed
  object is overwritten, and an object the source no longer has is deleted from the target. After
  a run the target matches the source, which is exactly what the verifier below checks.
- Is safely re-runnable: an interrupted transfer can just be re-run to continue where it left off,
  and a repeated run after the source has moved on converges instead of drifting.
- Streams each object directly from source to target (no local disk buffering), since source and
  target buckets are assumed to use different credentials/accounts.

Configure both buckets in `.env` (the source variables are the same ones used elsewhere in this
project; target variables are `TARGET_`-prefixed):

```bash
# Source bucket (existing variables, unchanged)
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-source-bucket"
AWS_ACCESS_KEY_ID="<source_access_key_id>"
AWS_SECRET_ACCESS_KEY="<source_secret_access_key>"

# Target bucket
TARGET_S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
TARGET_S3_BUCKET_NAME="my-target-bucket"
TARGET_AWS_ACCESS_KEY_ID="<target_access_key_id>"
TARGET_AWS_SECRET_ACCESS_KEY="<target_secret_access_key>"

S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
S3_FRESH_START="false"    # set true to delete the target's existing '<collection_id>/' prefix first
S3_DRY_RUN="false"        # set true to preview the copy plan without modifying the target
S3_DELETE_ORPHANS="true"  # set false for an additive-only copy that never removes target objects
```

Then run:

```bash
python script_s3_copy_source_bucket_collection_to_target_bucket_collection.py
```

`S3_FRESH_START` is the only way to trigger the destructive "clean the target path first" step
(e.g. for a first run against a target that already has stray/partial contents); it is never
triggered automatically. Omit it (or leave it `false`) for the default incremental,
resume-friendly sync.

Two safety rules apply whenever `S3_FRESH_START` is enabled:

- If the source listing turns up **no objects** for the collection (wrong bucket, wrong collection
  id, a misdetected layout), the run aborts with `S3OperationError` and deletes nothing — the
  target copy is never replaced with an empty one. This check also fires under `S3_DRY_RUN`.
- If the delete only **partially** succeeds, the run fails with `S3OperationError` rather than
  copying on top of a half-cleared prefix and reporting success.

Under `S3_DRY_RUN` the plan reports the full source object count as "to copy" (fresh_start clears
the target, so nothing is skipped) and logs how many objects the delete would remove.

`S3_DELETE_ORPHANS` (default `true`) controls the *incremental* path's own delete: target objects
under `<collection_id>/` that the source no longer has. Set it to `false` for an additive-only
copy — but note the verifier will then fail with `unexpected in target` after any source-side
deletion, because the two would no longer agree on what the collection contains. The same
empty-source-listing guard applies: if the source lists no objects, the run aborts with
`S3OperationError` rather than deleting the whole target copy as "orphaned" (this fires under
`S3_DRY_RUN` too), and a partial delete fails the run before any object is copied.

Every run logs its plan in four categories — to copy / to overwrite (changed in source) / to
delete (absent from source) / unchanged and skipped — so the log says why work happened.

> **Upgrading from 2.0.2 or earlier.** Previously the copier compared keys only, so it never
> re-copied a changed object and never removed a deleted one. A second copy of a collection whose
> `metadata/collection_config.json` had been written back to (the `ready_for_transfer` handshake)
> therefore failed verification permanently with a size mismatch, and a same-size content change
> left the target silently stale. Both are fixed. The behavioural change to be aware of is that
> `fresh_start=False` now deletes target-only objects unless you pass `delete_orphans=False`.

Folder uploads use object keys relative to the selected local folder. For example,
uploading `./EO.XXX.YYY.ZZZ/metadata/items` with no `target_path` uploads files like
`a.json` and `nested/b.json` at the bucket root. To place files under a collection or
metadata prefix in the bucket, pass that prefix explicitly as `target_path`.

### Verify a Collection Copy

`script_s3_verify_collection_copy.py` is a read-only smoke test to run right after the copy
script above completes: it uses `usergenerated.s3.s3_collection_copy_verifier.CollectionCopyVerifier`
to list the collection's objects in both buckets and confirms the target has exactly the same
relative keys as the source, with matching sizes for every key. Neither bucket is modified.

It reuses the same `.env` variables as the copy script above (no new variables needed):

```bash
python script_s3_verify_collection_copy.py
```

The script exits `0` and prints `Collection copy verification passed: True` when the target
matches the source. On a mismatch `CollectionCopyVerifier.run()` raises `S3OperationError`
whose message summarises the missing, unexpected, and/or size-mismatched keys (sampled, up to
20 per category); the script reports that one line and exits `1`.

Similarly, `CollectionCopier.run()` (the copy script above) raises `S3OperationError` — naming
the failed-object count and a sample of failed keys — if any object fails to copy. Objects
that did copy are left in place, so re-running resumes from where it stopped.

S3 helper code lives under `usergenerated/s3/`.

### Promote a Collection (regenerate → copy → verify → read, in one call)

`usergenerated.s3.collection_promoter.promote_collection` (also
`from usergenerated import promote_collection`) runs the whole "a collection has been staged
in a source bucket; validate it, copy it to a target bucket, and hand me its items" workflow
as a single call, in strict order:

1. `ItemGeneratorS3(...).run()` — regenerate STAC item metadata **in the source bucket**
   (source credentials; the `ready_for_transfer` safety gate stays on; `data/` is untouched).
2. `CollectionCopier(...).run()` — copy the collection source bucket → target bucket.
3. `CollectionCopyVerifier(...).run()` — confirm the target matches the source key-for-key.
4. Build a `PromotedCollection` read accessor over the **target** copy and call
   `on_ready(handle)` **exactly once**.

`on_ready` is a callback you provide. Its argument exposes just two methods:

- `handle.read_collection() -> dict` — the target copy's `collection.json`
  (wraps `get_collection_from_s3`).
- `handle.iter_items() -> Iterator[dict]` — the target copy's STAC Items, streamed one at a
  time (wraps `iter_collection_items_from_s3`; lazy — consume it inside the callback).

Any domain-specific downstream work (database load, re-publish, indexing) stays entirely in
your `on_ready`; `promote_collection` owns no datastore and never returns a status — a normal
return means every stage passed and `on_ready` ran.

- `on_ready` runs **only after copy and verify have both succeeded**. Any earlier failure
  raises (`S3OperationError` / `ConfigError` / `CredentialsError` / `FileNotFoundError`, and
  `FileExistsError` from regeneration) and `on_ready` is never called.
- `dry_run=True` runs regeneration/validation only (via `ItemGeneratorS3(dry_run=True)`); the
  copier and verifier are never constructed and `on_ready` is never called.
- `fresh_start=True` passes through to `CollectionCopier`, deleting the target
  `<collection_id>/` prefix before copying. The source bucket is never modified. It aborts without
  deleting if the source listing is empty, and fails the run if the delete only partly succeeds.
- `delete_orphans=True` (the default) passes through to `CollectionCopier`: the copy removes
  target objects the source no longer has, so a repeat promotion after items were dropped or
  rewritten still verifies. Pass `delete_orphans=False` for an additive-only copy.
- `asset_href_prefix="..."` (optional) is used for the regenerated item asset hrefs, overriding any
  `asset_href_prefix` in the source `collection_config.json` or an `item_config.json`; `""` forces no
  prefix. The source config is not modified. See
  [Overriding the prefix at run time](#overriding-the-prefix-at-run-time).
- Idempotent / resumable: re-running after a failure repeats the skip-aware copy, re-verifies,
  and re-invokes `on_ready`. Side effects are only on the source (item regen, plus a
  timestamped `items` backup when `backup_items=True`) and the target (the copy).

```python
from usergenerated import PromotedCollection, promote_collection


def on_ready(collection: PromotedCollection) -> None:
    stac = collection.read_collection()
    item_count = sum(1 for _ in collection.iter_items())
    print(f"{stac['id']}: {item_count} item(s) ready in the target bucket")


promote_collection(
    collection_id="EO.XXX.YYY.ZZZ",
    source_endpoint_url="https://s3.central.data.destination-earth.eu",
    source_bucket_name="my-source-bucket",
    source_aws_access_key_id="<source_access_key_id>",
    source_aws_secret_access_key="<source_secret_access_key>",
    target_endpoint_url="https://s3.central.data.destination-earth.eu",
    target_bucket_name="my-target-bucket",
    target_aws_access_key_id="<target_access_key_id>",
    target_aws_secret_access_key="<target_secret_access_key>",
    on_ready=on_ready,
)
```

`script_s3_promote_collection.py` is a runnable `.env`-driven demo whose `on_ready` logs the
promoted collection's id and item count. It reuses the same variables as the copy and
regenerate scripts (`S3_COLLECTION_ID`, `S3_ENDPOINT_URL`, `S3_BUCKET_NAME`,
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `TARGET_S3_ENDPOINT_URL`,
`TARGET_S3_BUCKET_NAME`, `TARGET_AWS_ACCESS_KEY_ID`, `TARGET_AWS_SECRET_ACCESS_KEY`,
`S3_DRY_RUN`, `S3_FRESH_START`), plus optional keys `S3_LOCAL_ROOT` (default `tmp_structure_mirror`),
`S3_BACKUP_ITEMS` (default `true`) and `S3_ASSET_HREF_PREFIX` (unset = use the configured prefix; set to
empty = force no prefix).

```bash
python script_s3_promote_collection.py
```

### Manipulate Collection Config in S3

`script_s3_collection_config_manipulation.py` demonstrates the helpers in
`usergenerated.s3.s3_collection_config_tools` for reading and updating a collection's
`collection_config.json` directly in S3, without a local mirror. Unlike the verify script above, it
**mutates** the live bucket. For a given collection it:

1. Resolves the config key with `resolve_collection_config_key` and prints the full `collection_config.json`.
2. Reads the `ready_for_transfer` flag via `get_ready_for_transfer`.
3. Sets `ready_for_transfer` to `false` via `set_ready_for_transfer`.
4. Adds an ad-hoc `last_transferred` field (current UTC time, ISO 8601) via `update_collection_config`.

Setting `ready_for_transfer` to `false` engages the `regenerate-item-metadata-s3` safety gate described
in [Safety Behavior](#safety-behavior) — a subsequent regeneration run will refuse to proceed until the
flag is set back to `true` (or the check is explicitly skipped).

For programmatic use the same module exposes two reusable building blocks so a caller does not have to
chain layout detection and a hand-written read-modify-write:

- `resolve_collection_config_key(s3_client, bucket_name, collection_id)` — locate `collection_config.json`
  in the bucket without handling `BucketLayout` yourself.
- `update_collection_config(s3_client, bucket_name, key, changes)` — one read-modify-write for an
  arbitrary set of fields. `changes` is either a mapping merged into the config (use the exported
  `REMOVE` sentinel as a value to delete a key) or a callback that mutates the dict in place. Keys you
  do not touch keep their value, order, and the file's `indent=4` formatting; it raises `ConfigError`
  if the object is missing or not a JSON object and `S3OperationError` on an S3 read/write failure.
  `set_ready_for_transfer` is a one-line wrapper over it.

Configure the target bucket in `.env` (the same single-bucket variables used by the regeneration demo
script):

```bash
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
```

Then run:

```bash
python script_s3_collection_config_manipulation.py
```

### Report Bucket Information

`script_s3_bucket_information.py` demonstrates `usergenerated.s3.s3_bucket_info.get_bucket_information`,
which makes a single paginated sweep of a bucket (or a prefix within it) and prints a JSON report:
file count, total and average size, the 5 largest files, oldest/most-recent modification times, a
count+size breakdown by top-level folder, a `generated_at` timestamp, and a sample file listing
capped at 50 entries (with a `sample_truncated` flag) so you never get thousands of rows back.
Every size in the report (`total_size_bytes`, `average_file_size_bytes`, and each `size_bytes` in
`largest_files`, `top_level_prefixes`, and `sample_files`) has a `..._human`/`size_human`
counterpart, e.g. `"total_size_human": "94.2 MB"`. It is **read-only** — the bucket is never
modified.

Writing the report to `output/<YYYYMMDDTHHMMSS>_Bucket_Information.json` (the timestamp matches
`generated_at` in the JSON) is **off by default** — the report is only printed to stdout. Turn it
on with either `S3_WRITE_OUTPUT_FILE="true"` in `.env`, or `write_output_file=True` when calling
`run_get_bucket_information` programmatically; the `output/` directory is created if needed.

Configure the bucket in `.env` (the same single-bucket variables used by the config manipulation
script above); `S3_PREFIX`, `S3_BUCKET_INFO_SAMPLE_LIMIT`, `S3_WRITE_OUTPUT_FILE`, and
`S3_BUCKET_INFO_OUTPUT_DIR` are optional:

```bash
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_PREFIX=""                        # optional: scope the report to one folder/collection
S3_BUCKET_INFO_SAMPLE_LIMIT="50"    # optional: cap on the sample file listing
S3_WRITE_OUTPUT_FILE="false"        # optional: set true/1/yes/on to also write the JSON report to disk
S3_BUCKET_INFO_OUTPUT_DIR="output"  # optional: where the timestamped JSON report is written
```

Then run:

```bash
python script_s3_bucket_information.py
```

### Copy Collection Metadata Locally

`script_s3_get_metadata.py` demonstrates
`usergenerated.s3.s3_collection_metadata_downloader.download_collection_metadata`, which copies a
collection's `metadata/` folder from S3 to a local destination. The source bucket's layout
(whether it holds `<collection_id>/metadata/` or a top-level `metadata/`) is detected
automatically, so the caller never needs to know or check it beforehand — the result always lands
under a folder named for the collection ID, e.g. `<destination_root>/EO.XXX.YYY.ZZZ/metadata`. It
is **read-only** with respect to S3 — nothing in the source bucket is modified.

The destination root is configurable via `S3_METADATA_DESTINATION_ROOT` (or the
`destination_root` argument when calling `run_get_metadata` programmatically); when unset, it
defaults to a fresh timestamped folder under the OS temp directory (`/tmp` on Linux, `%TEMP%`
on Windows), e.g. `<os-temp-dir>/usergenerated_tmp/20260824T101500/EO.XXX.YYY.ZZZ/metadata`.

Objects are downloaded concurrently — one worker thread per object, bounded by the
`max_workers` argument (default `MAX_METADATA_DOWNLOAD_WORKERS` = 16, capped at 32). The single
`s3_client` is shared across those threads, so build it with a matching connection pool:
`create_s3_client(..., max_pool_connections=MAX_METADATA_DOWNLOAD_WORKERS)` (botocore defaults
to 10). If any object fails to download the whole call raises `S3OperationError` (a
`RuntimeError` subclass) rather than returning a partial folder; once a systemic problem has
failed ~20 objects it stops submitting
new downloads instead of grinding through every remaining key.

For callers that only need the files for the duration of a block — parse `collection.json`,
stream `items/` somewhere, done — use the `downloaded_collection_metadata` context manager
instead of calling `download_collection_metadata` directly; on exit it removes the
`<collection_id>/` subtree it downloaded, and if it also generated the timestamped destination
root (no explicit `destination_root` given) it removes that too, so nothing is left under
`/tmp`. An explicit `destination_root` you pass in is otherwise left in place:

```python
with downloaded_collection_metadata(s3_client, bucket, collection_id) as metadata_folder:
    collection = get_stac_collection(metadata_folder)
    for item in iter_items(metadata_folder):
        ...
# the timestamped <os-temp-dir>/usergenerated_tmp/<...> directory is gone here
```

The same module also provides small supporting functions for reading the downloaded metadata:
`get_stac_collection`/`get_collection_config` load `collection.json`/`collection_config.json` as
dicts, and `iter_items`/`iter_item_batches` stream STAC Items out of the downloaded `items/`
folder — one item at a time, or in fixed-size batches — rather than loading every item into
memory at once. This matters for collections with very large item counts (hundreds of thousands),
where materializing every item as a single list would be memory-intensive. The batch size defaults
to `METADATA_DOWNLOADER_DEFAULT_ITEM_BATCH_SIZE` (500) and is configurable via
`S3_METADATA_ITEM_BATCH_SIZE` (or the `item_batch_size` argument to `run_get_metadata`).

If you only need to walk the items once and don't need them to persist,
`iter_collection_items_from_s3` / `iter_collection_item_batches_from_s3` (and
`get_collection_from_s3` for `collection.json`) stream STAC Items straight from the bucket with no
local copy written and no cleanup to do — a drop-in for `iter_items` / `iter_item_batches` /
`get_stac_collection`. The bucket layout is detected automatically, item bodies are fetched with a
bounded concurrent prefetch (`max_workers` defaults to `MAX_METADATA_DOWNLOAD_WORKERS`, capped at
32), and any fetch or parse failure raises `S3OperationError` rather than yielding a partial
result. Keep `download_collection_metadata` for when the files must stay on disk or be read more
than once. Setting `S3_METADATA_STREAM_ITEMS` to a truthy value (or passing `stream_items=True` to
`run_get_metadata`) runs `script_s3_get_metadata.py` on this no-download path.

Configure the collection and bucket in `.env` (the same single-bucket variables used by the config
manipulation script above); `S3_METADATA_DESTINATION_ROOT`, `S3_METADATA_ITEM_BATCH_SIZE`, and
`S3_METADATA_STREAM_ITEMS` are optional:

```bash
S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_METADATA_DESTINATION_ROOT=""  # optional: local folder to copy metadata/ into (default: <os-temp-dir>/usergenerated_tmp/<timestamp>)
S3_METADATA_ITEM_BATCH_SIZE=""  # optional: items per batch when streaming items/ (default: 500)
S3_METADATA_STREAM_ITEMS=""  # optional: truthy = skip the local download, stream items straight from S3
```

Then run:

```bash
python script_s3_get_metadata.py
```

## Inspect a Deployed Collection

A collection is **deployed** once it is served by the DestinE Data Lake HDA STAC API at
`https://hda.data.destination-earth.eu/stac/v2/collections`. This is the check that comes *after* a
collection has been promoted to the target bucket and picked up by DEDL.

Some collections are protected by a role and are simply absent for anonymous callers, so
`script_deployed_information.py` always makes **both** attempts and reports them side by side:

- **anonymous** — a plain request, no credentials.
- **authenticated** — a request carrying a DEDL access token as an `Authorization: Bearer` header.
  The token comes from a DESP username/password via the [`destinelab`](https://pypi.org/project/destinelab/)
  token exchange, and makes the collections your account is entitled to see visible.

From the two outcomes it derives:

- `deployed` — either attempt found the collection.
- `requires_authentication` — only the authenticated attempt found it, i.e. the collection is
  deployed but protected.

The script is **read-only**: it issues HTTP GETs only, and writes nothing to the STAC API or to S3.
The DESP password and the DEDL token never appear in the report or in the logs.

### Install the optional dependency

The authenticated attempt needs `destinelab`, which ships as an optional extra so anonymous lookups
work on a plain install:

```bash
uv sync --extra deployed
```

Without it (or without DESP credentials), the authenticated attempt is reported as skipped and the
run still succeeds with the anonymous result.

### Configuration

Add the keys you need to `.env`:

```env
DEPLOYED_COLLECTION_ID="EO.XXX.YYY.ZZZ"
DESP_USERNAME="<desp_username>"  # optional: omit to run the anonymous lookup only
DESP_PASSWORD="<desp_password>"  # optional: omit to run the anonymous lookup only
DEPLOYED_STAC_BASE_URL=""  # optional: STAC API base URL (default: https://hda.data.destination-earth.eu/stac/v2)
DEPLOYED_AUTHENTICATE=""  # optional: falsy = skip the authenticated attempt entirely (default: true)
DEPLOYED_OUTPUT_DIR=""  # optional: folder for the output file (default: output)
DEPLOYED_WRITE_OUTPUT_FILE=""  # optional: falsy = skip writing the collection file (default: written)
```

The output file contains **only the STAC Collection** — not the surrounding report — so it can be
diffed straight against a local `metadata/collection.json`. Before writing, the document is loaded
into PySTAC and validated:

- **Normalized** — the file is PySTAC's own serialization, so its field ordering is standardized
  rather than whatever the server happened to return, and successive runs produce diffable files.
  Extension and vendor fields (`dedl:*`, `cube:*`, `sci:*`, …) are preserved.
- **Validated** — the result is reported in the console report as `stac_valid` and
  `stac_validation_error`, and the report also gains the `output_file` path. Validation is
  *reported, not enforced*: a deployed collection that fails validation is still written, since it
  is not yours to fix and is exactly the file worth inspecting. A document PySTAC cannot read as a
  Collection at all raises `CollectionValidationError`.

The filename carries a filename-safe form of the collection id (every run of non-alphanumeric
characters collapses to a single underscore, so `EO.XXX.YYY.ZZZ` becomes `EO_XXX_YYY_ZZZ`), so
several collections can share one output folder:

```
output/20260904T164512_EO_XXX_YYY_ZZZ_Deployed_Collection.json
```

When both attempts found the collection the authenticated body is written (it is the only one that
can see a protected collection; for a public one the two are identical). Nothing is written when
neither attempt found it.

Then run:

```bash
python script_deployed_information.py
```

Example output for a deployed but role-protected collection (the full STAC Collection body is
included under each successful attempt, truncated here):

```json
{
    "collection_id": "EO.XXX.YYY.ZZZ",
    "base_url": "https://hda.data.destination-earth.eu/stac/v2",
    "anonymous": {
        "attempted": true,
        "found": false,
        "status_code": 404,
        "error": "Collection 'EO.XXX.YYY.ZZZ' is not visible to this caller (HTTP 404).",
        "collection": null
    },
    "authenticated": {
        "attempted": true,
        "desp_username": "user@example.org",
        "roles": ["..."],
        "found": true,
        "status_code": 200,
        "error": null,
        "collection": { "type": "Collection", "id": "EO.XXX.YYY.ZZZ", "...": "..." }
    },
    "deployed": true,
    "requires_authentication": true,
    "generated_at": "2026-09-04T16:23:14.859202+00:00"
}
```

### Programmatic Usage

The helpers behind the script live in `usergenerated.deployed` and can be used directly:

```python
from usergenerated.deployed.auth import authenticate_dedl
from usergenerated.deployed.stac_api import get_collection, list_collection_ids

# Anonymous - only public collections are visible.
lookup = get_collection("EO.ESA.DAT.SENTINEL-2.MSI.L1C")
print(lookup.found, lookup.status_code)

# Authenticated - protected collections the account is entitled to see become visible.
authentication = authenticate_dedl("<desp_username>", "<desp_password>")
protected = get_collection("EO.XXX.YYY.ZZZ", token=authentication.token)
print(authentication.roles, protected.collection)

# Every collection id visible to this caller.
print(len(list_collection_ids(token=authentication.token)))
```

`get_collection` returns a `CollectionLookup` (`found`, `status_code`, `collection`, `error`).
A collection that is not visible is reported as `found=False` with the HTTP status - not an
exception. Transport failures, unexpected statuses, and unparseable bodies raise `DeployedApiError`;
missing DESP credentials or an unusable token raise `CredentialsError`; a missing `destinelab`
install raises `ConfigError` telling you to run `uv sync --extra deployed`.

### Compare a Deployed Collection with the Local `collection.json`

`script_compare_collections.py` checks that two STAC Collection files hold the same content — typically
the deployed collection written above against the `metadata/collection.json` it was published from:

```bash
python script_compare_collections.py \
    output/20260914T142622_EO_FMI_DAT_DESTINE_CLIMATE_WILDFIRE_FWI_Deployed_Collection.json \
    EO.FMI.DAT.DESTINE_CLIMATE_WILDFIRE_FWI/metadata/collection.json
```

- **Key order is irrelevant.** Both files are loaded through pystac first, so an unsorted, hand-written
  `collection.json` compares on equal terms with the pystac-sorted deployed file. List order still
  counts (`bbox`, `keywords`, `providers`).
- **Server-managed fields are ignored by default:** `links`, `created`, `updated`, `published`,
  `federation` and `summaries/federation:backends`. A container left empty by that (e.g. `summaries`)
  counts as absent.
- **`stac_extensions` is always compared.** Every extension the deployed collection declares must also
  be declared in `collection.json` — including the timestamps extension
  (`https://stac-extensions.github.io/timestamps/v1.1.0/schema.json`), which the deployed collection
  carries because of its `created`/`updated`/`published` fields.
- `--ignore FIELD` adds a field to ignore, as a `/`-separated path from the document root (e.g.
  `--ignore assets/thumbnail`); `--ignore-extension URI_PREFIX` adds a `stac_extensions` prefix;
  `--no-default-ignores` compares everything. Both flags are repeatable.

It prints a JSON report listing **every** difference with its path (e.g. `extent/spatial/bbox[0][2]`,
`providers[1]`) and kind (`value_mismatch`, `only_in_reference`, `only_in_candidate`), then exits `0`
when the collections match and `1` when they differ, so it can gate a pipeline. It reads the two files
only; nothing is written and no network request is made.

Programmatically:

```python
from pathlib import Path
from usergenerated.deployed.collection_comparator import compare_collection_files

comparison = compare_collection_files(Path("deployed.json"), Path("collection.json"))
for difference in comparison.differences:
    print(difference.path, difference.kind.value, difference.reference, difference.candidate)
```

## Runtime Initialization

Logging initialization is now explicit and idempotent.

1. Entry-point scripts call `setup_logging()` from `usergenerated.logging_config` at startup.
2. Environment loading (`load_dotenv`) is also called explicitly at startup.
3. Importing modules should not configure logging implicitly.

This keeps behavior stable regardless of import order and avoids duplicate log handlers.

### Windows

The package is tested on Linux and Windows (CI runs the suite on both). A few platform notes:

- **Temp directory.** Features that mirror or download from S3 (`regenerate-item-metadata-s3`,
  `script_s3_get_metadata.py`) use the OS temp directory (`%TEMP%` on Windows) for their working
  copy, not a hardcoded `/tmp`. Override it with `S3_METADATA_DESTINATION_ROOT` where supported.
- **Content-based MIME sniffing.** `itemhelper.guess_mime_type_advanced()` needs the optional
  `magic` extra (`uv sync --extra magic` / `pip install "usergenerated[magic]"`) and a native
  `libmagic`, which is not present on a default Windows install. Nothing in the standard workflow
  calls it — media types are inferred from the file extension.
- **Symlinks in `data/`.** Creating symlinks on Windows needs `SeCreateSymbolicLinkPrivilege`
  (Developer Mode or an elevated shell). The tools skip symlinks on upload regardless.
- **S3 keys with unusual names.** Keys containing characters that are illegal in Windows
  filenames (`: * ? " < > |`), or that differ only in case, are sanitised to a safe local name
  when mirrored; the original key is preserved in the per-folder manifest.
- **Long paths (the 260-character limit).** Windows refuses to create a path longer than 260
  characters unless long-path support is switched on. This is reachable in practice: an item
  folder name embeds the whole collection id (`<collection_id>_<start>_<end>`), so a deep
  working directory plus a long collection id can cross the limit while mirroring. The tools
  detect this before creating anything and fail with a message naming the path and its length,
  rather than the misleading `WinError 3: The system cannot find the path specified`. Fix it by
  running from a shallower directory, passing a shorter `--local-root` /
  `S3_METADATA_DESTINATION_ROOT`, or enabling long paths system-wide:

  ```powershell
  # Run as Administrator, then reboot.
  Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' `
      -Name LongPathsEnabled -Value 1
  ```

- **Collection folder name case.** The Windows filesystem is case-insensitive, so a folder named
  `eo.xxx.yyy.zzz` would otherwise satisfy a request for `EO.XXX.YYY.ZZZ` and then fail for a
  colleague on Linux. Collection ids are case-sensitive, so this is reported as an error on every
  OS — rename the folder to match the id exactly.
- **Console output.** Log records and progress bars are written as UTF-8 with a safe fallback, so
  a non-ASCII filename or collection description does not break a legacy `cmd.exe` console.

### Exit codes and failure output

Every console command (`generate-item-metadata`, `regenerate-item-metadata-s3`) and root
`script_s3_*.py` shares one convention:

| Exit code | Meaning |
|---|---|
| `0` | success |
| `1` | an expected failure (bad config, missing credentials, a missing file or folder you named, invalid `collection.json`, a failed S3 operation) — reported as a single `ERROR` log line, no traceback; or an unexpected internal error, which keeps its traceback |
| `2` | command-line usage error (bad or missing arguments) |

## Troubleshooting

1. Generation fails with folder-level errors:
Set `item_folder_level` to match your actual `data/` layout.

2. Generation fails with "non-conforming path(s) under 'data/'":
The structure conformance check rejected your layout. Each reported line names the offending path
and the remedy - see [Structure Conformance Check](#structure-conformance-check).

3. Missing `item_config.json` errors:
Set `item_config_optional` to `true` if item config files are intentionally omitted.

4. Collection id validation errors:
Ensure root folder name and `collection_config.json > id` represent the same collection id.

5. Upload credential errors:
Ensure `.env` contains valid `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` when upload is enabled.

## External References

1. [DestinE Data Lake - Promote User Data to become DestinE data](https://destine-data-lake-docs.data.destination-earth.eu/en/latest/dedl-discovery-and-data-access/User-Generated-Data/Promote-user-data-to-become-DestinE-data/Promote-user-data-to-become-DestinE-data.html)
2. [STAC Collection specification](https://stacspec.org/en/about/stac-spec/)
3. [STAC Item specification](https://stacspec.org/en/about/stac-spec/)
