Metadata-Version: 2.4
Name: biomero-importer
Version: 1.5.0b5
Summary: BIOMERO.importer - Automated Data Import System (formerly OMERO ADI)
Author-email: Core Facility - Cellular Imaging <cellularimaging@amsterdamumc.nl>
Project-URL: Homepage, https://github.com/Cellular-Imaging-Amsterdam-UMC/BIOMERO.importer
Project-URL: Documentation, https://cellular-imaging-amsterdam-umc.github.io/NL-BIOMERO/
Project-URL: Repository, https://github.com/Cellular-Imaging-Amsterdam-UMC/BIOMERO.importer
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: License :: OSI Approved :: GNU General Public License v2 (GPLv2)
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENCE
Requires-Dist: ezomero==3.2.3
Requires-Dist: pandas>=2.2.1
Requires-Dist: numpy>=1.26.4
Requires-Dist: openpyxl>=3.1.2
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pytest>=8.2.1
Requires-Dist: sqlalchemy>=2.0.28
Requires-Dist: psycopg2>=2.9.7
Requires-Dist: zarr>=3
Requires-Dist: alembic>=1.13
Requires-Dist: biomero-schema<0.3,>=0.2
Provides-Extra: test
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: mock; extra == "test"
Requires-Dist: psycopg2-binary; extra == "test"
Provides-Extra: identity
Requires-Dist: iscc-bio[ome-zarr]<0.3,>=0.2; extra == "identity"
Dynamic: license-file

# BIOMERO.importer - Automated Data Import System
[![Build BIOMERO.importer Docker Image](https://github.com/NL-BioImaging/BIOMERO.importer/actions/workflows/docker-image.yml/badge.svg)](https://github.com/NL-BioImaging/BIOMERO.importer/actions/workflows/docker-image.yml) [![Publish BIOMERO.importer to PyPI](https://github.com/NL-BioImaging/BIOMERO.importer/actions/workflows/publish-to-pypi.yml/badge.svg)](https://github.com/NL-BioImaging/BIOMERO.importer/actions/workflows/publish-to-pypi.yml) [![Python package](https://github.com/NL-BioImaging/BIOMERO.importer/actions/workflows/python-package.yml/badge.svg)](https://github.com/NL-BioImaging/BIOMERO.importer/actions/workflows/python-package.yml)
> 🚀 **This package is part of <img src="https://raw.githubusercontent.com/NL-BioImaging/OMERO.biomero/refs/tags/v1.2.1/webapp/src/img/biomero-logo.svg" alt="BIOMERO" height="16" style="height:16px; width:auto; vertical-align:middle;"> BIOMERO 2.0** — For complete deployment and FAIR infrastructure setup, start with the [**NL-BIOMERO Documentation**](https://nl-bioimaging.github.io/NL-BIOMERO/) 📖

The BIOMERO.importer system enables automated uploading of image data from microscope workstations to an OMERO server. BIOMERO.importer is a database-driven system that polls a PostgreSQL database for new import orders and processes them automatically, including the option of running preprocessing containers for e.g. file conversion or pyramid creation.

## System Overview

The BIOMERO.importer system consists of:

1. **Database-driven order management**: Upload orders are stored in a PostgreSQL database with full tracking and preprocessing support
2. **Automated polling**: The system continuously polls the database for new orders to process
3. **Ingestion pipeline**: Handles file validation, optional preprocessing, and OMERO import with comprehensive logging
4. **Event sourcing**: All import steps are tracked in the database for full auditability

## Architecture

The system uses SQLAlchemy models to manage:

- **Upload Orders**: Stored in `imports` table with stages from "Import Pending" to "Import Completed"
- **Preprocessing**: Optional containerized preprocessing steps stored in `imports_preprocessing` table
- **Progress Tracking**: Complete audit trail of all import operations

![Flow Diagram of BIOMERO.importer process](flow_diagram_ADI_import.png)

### Key Components

- **DatabasePoller**: Continuously polls for new orders with `STAGE_NEW_ORDER` status
- **UploadOrderManager**: Validates and processes order data from database records
- **DataPackageImporter**: Handles the actual OMERO import process with optional preprocessing
- **IngestTracker**: Manages database logging and progress tracking

## Database Schema

The system uses two main tables:

### `imports` (IngestionTracking)
- Stores all import orders and their progress
- Tracks stages: "Import Pending" → "Import Started" → "Import Completed"/"Import Failed"
- Includes full metadata: user, group, destination, files, timestamps
- Stores optional Zarr registration choices in the nullable `import_options`
  JSON text column. Existing orders without it behave exactly as before.

### `imports_preprocessing` 
- Stores preprocessing configuration for containerized workflows
- Links to imports records via foreign key
- Supports dynamic parameters via JSON field

## Configuration

Configure the system using `config/settings.yml`:

```yaml
# Database connection (can also be set via INGEST_TRACKING_DB_URL environment variable)
ingest_tracking_db: "postgresql://user:password@host:port/database"

# OMERO connection (set via environment variables)
# OMERO_HOST, OMERO_USER, OMERO_PASSWORD, OMERO_PORT

# File system paths (legacy - only base_dir is used in current implementation)
base_dir: /data

# Processing settings
max_workers: 4
log_level: DEBUG
log_file_path: logs/app.logs

# Import optimization
parallel_upload_per_worker: 2
parallel_filesets_per_worker: 2
skip_checksum: false
skip_minmax: false
skip_thumbnails: false
skip_upgrade: false
skip_all: false

use_register_zarr: true

# Annotation namespace for OMERO metadata (default: "biomero.import")
# Can be customized to maintain compatibility with existing systems
annotation_namespace: "biomero.import"
```

**Note**: The `upload_orders_dir_name`, `data_dir_name`, and `failed_uploads_directory_name` settings are **legacy from the old file-based system** and are no longer used in the current database-driven implementation.

## Environment Variables

The system uses these environment variables:

- `INGEST_TRACKING_DB_URL`: Database connection string (overrides config file setting)
- `OMERO_HOST`: OMERO server hostname
- `OMERO_USER`: OMERO root user
- `OMERO_PASSWORD`: OMERO root password
- `OMERO_PORT`: OMERO server port
- `PODMAN_USERNS_MODE`: Set to "keep-id" for Linux user namespace mapping in preprocessing
- `PROCESSED_DATA_FOLDER`: Subfolder name for preprocessing outputs and canonical
  Zarr storage (default: `.processed` when the variable is unset). Read when the
  importer starts. The value is used as supplied: no leading dot is added or
  removed. Use a non-empty relative subfolder name.
- `USE_REGISTER_ZARR`: Set to "true" to enable zarr register script - requires omero-zarr-pixel-buffer (overrides config file setting)
- `BIOMERO_SHALLOW_ZARR`: Opt in to the native `biomero.shallow-zarr`
  lifecycle operation. Existing orders are unchanged when false or absent.
  Enabling it requires installing the importer with its identity extra:
  `pip install "biomero-importer[identity]"`. The NL-BIOMERO importer image
  includes this extra. If the flag is enabled without ISCC-BIO,
  `get_importer_capabilities()` omits the lifecycle operation, reports the
  missing dependency, and rejects shallow import orders with an actionable
  configuration error; ordinary import orders remain available.
- `BIOMERO_SHALLOW_ZARR_WORKERS`: Bounded ISCC-BIO identity workers used by
  the importer service (library fallback `1`; NL-BIOMERO supplies `4`). This is
  deployment configuration, not a client-controlled import option.

### Configuring the processed data folder

`PROCESSED_DATA_FOLDER=processed` writes to a subfolder named `processed`;
`PROCESSED_DATA_FOLDER=.import` writes to `.import`. Leaving the variable unset
keeps the existing `.processed` default.

> [!WARNING]
> **Choose this setting before the first import whenever possible. Changing it
> on an existing deployment requires planning; there is no built-in migration.**
>
> Changing from `.processed` to `.import` leaves existing data in `.processed`
> and directs new processed outputs to `.import`, so both folders can coexist.
> The importer does not move existing data, retarget existing filesystem
> symlinks, rewrite Zarr `ExternalInfo.lsid` values, or migrate canonical-source
> records. Changing this variable alone does not redirect existing pixel reads.
> **Keep the old data accessible at its original paths from OMERO.server.**
> Renaming, moving, or deleting the old folder can break both symlink-based
> imports and registered Zarr images, including their Plate images.
>
> Resubmitting an order with preprocessing runs the converter with the current
> folder setting. CSV annotation lookup checks the input directory and the
> currently configured processed subfolder; it does not search the previous
> processed subfolder. Let active imports finish before changing the setting,
> retain the old folder and its mounts, and plan any data migration separately.

The storage references explain why existing reads retain their paths:

| Import route | Stored reference used for pixel access |
| --- | --- |
| Conventional in-place import | A filesystem symlink in OMERO's ManagedRepository. For preprocessed files, `upload_files()` retargets it to the converter's reported `full_path`, mapped to shared storage. |
| Zarr registration (`USE_REGISTER_ZARR=true`) | `set_external_info()` records the Zarr path in each Image's `ExternalInfo.lsid`, appending the image node for series and Plates. The Zarr pixel buffer reads that stored path. This route does not rely on the ManagedRepository symlink redirection. |
| Shallow Zarr registration | The importer resolves the stored canonical source or label location before registration, then records that physical path in `ExternalInfo.lsid`. Existing canonical `relativePath` values are resolved against their storage root, without substituting the current processed-folder setting. |

See [`upload_files()`](biomero_importer/utils/importer.py),
[`set_external_info()`](biomero_importer/utils/register.py), and
[`resolve_managed_source_path()`](biomero_importer/utils/result_zarr.py).
In the pixel buffer, `ZarrPixelsService.getUri()` reads `ExternalInfo.lsid`,
`asPath()` converts a local value with `Paths.get()`, and
`createOmeNgffPixelBuffer()` opens that location. It does not read
`PROCESSED_DATA_FOLDER` or search for a renamed folder. See the upstream
[0.6.1 implementation](https://github.com/glencoesoftware/omero-zarr-pixel-buffer/blob/v0.6.1/src/main/java/com/glencoesoftware/omero/zarr/ZarrPixelsService.java)
and [path contract](https://github.com/glencoesoftware/omero-zarr-pixel-buffer/blob/v0.6.1/README.md#usage).

The setting is local to each Python process importing the library. Setting it
on the importer container configures that service's preprocessing. Code calling
`CanonicalStore.relative_path_for()` uses the setting in its own process;
setting it only on the importer does not configure separate OMERO script
workers. If those workers must create canonical Zarrs in the same custom folder,
their deployment must also pass the variable through to the script subprocesses.

For example, set a different processed subfolder in the importer container's
Docker Compose environment, then recreate the service:

```yaml
services:
  biomero-importer:
    environment:
      PROCESSED_DATA_FOLDER: processed
```

```bash
docker compose up -d --force-recreate biomero-importer
```

## Creating Upload Orders

Upload orders are typically created through a user interface, such as the OMERO.biomero plugin (Importer tab) at `/omero_biomero/biomero/`, an OMERO.web extension. However, orders can also be created programmatically. New integrations should call `biomero_importer.submit_import_order(order)` and inspect `biomero_importer.get_importer_capabilities()` before requesting an optional lifecycle operation. The API validates and writes the same append-only database order used by existing clients; direct legacy database writers remain supported.

You can use the provided test scripts shown below as examples. 
You can also configure some more settings for them: 
```yaml
# Preprocessing settings
preprocessing: true  # Enable containerized preprocessing
sample_image: /auto-importer/tests/Barbie.tif
sample_group: "Demo"
sample_user: "researcher"
sample_parent_id: "151"
sample_parent_type: "Dataset"  # or "Screen"
```

### BIOMERO shallow Plate registration

Native importer lifecycle operations are carried in the versioned
`ImportOptionsEnvelope` from `biomero-schema`. They execute after any existing
container preprocessing and before OMERO registration. This means a direct
Zarr or a Zarr produced by the existing converter can request the same
post-processing behavior. The lifecycle engine returns a registration plan and
does not depend on `register.py`; a future OMERO CLI Zarr importer can consume
the same plan.

Empty options and the earlier flat schema-1 registration options are upcast to
an envelope with no operations, so they continue through the established
import path. `imports_preprocessing` remains the legacy external-container
contract and is not repurposed for native operations.

BIOMERO workflow result orders may include an `ImportOptions` object defined by
`biomero-schema`. For a managed shallow Plate, the default
`{"platePixelSource":"source"}` keeps the Plate/Well/WellSample hierarchy and
registers each child Image against the managed canonical Plate pixels. The
optional `{"platePixelSource":"label","plateLabelName":"nuclei"}` registers
the same hierarchy against that label under every image node, producing a
mask-backed Plate view without copying arrays. BIOMERO creates these orders;
ordinary importer clients can omit `ImportOptions`.

BIOMERO-generated workflow inputs may carry `.biomero-input.json`, containing
one serialized `CanonicalInput`. The importer accepts it only when it exactly
matches the authoritative `CanonicalInputManifest` in the lifecycle operation,
then independently verifies the returned pixels. This disambiguates renamed
outputs derived from separately selected Images with identical content. The
temporary marker is consumed before registration; missing markers retain the
existing artifact-name and content-identity fallback.

The shallow collection remains the authoritative in-place result and receives
a compact Plate-level OMERO annotation. Label-backed registration requires the
named label to exist on every Plate image and fails rather than guessing when
the selection is incomplete or ambiguous.

### Using the System Check Script

```bash
# Inside the container
python tests/system_check.py
```

This script creates a test upload order and verifies the complete ingestion pipeline.

### Using the Test Main Script

```bash
# Inside the container  
python tests/t_main.py
```

This creates upload orders for multiple groups based on your configuration.

### Manual Database Insertion

```python
from biomero_importer.utils.ingest_tracker import IngestionTracking, Preprocessing, STAGE_NEW_ORDER
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine

# Create database connection
engine = create_engine("postgresql://user:password@host:port/database")
Session = sessionmaker(bind=engine)
session = Session()

# Create basic upload order
order = IngestionTracking(
    group_name="Demo",
    user_name="researcher",
    destination_id="151",
    destination_type="Dataset", 
    stage=STAGE_NEW_ORDER,
    uuid=str(uuid.uuid4()),
    files=["/data/group/image1.tif", "/data/group/image2.tif"]
)

# Optional: Add preprocessing
preprocessing = Preprocessing(
    container="cellularimagingcf/converter:latest",
    input_file="{Files}",
    output_folder="/data",
    alt_output_folder="/out",
    extra_params={"saveoption": "single"}
)
order.preprocessing = preprocessing

session.add(order)
session.commit()
session.close()
```

## Preprocessing Support

The system supports containerized preprocessing workflows using **Podman-in-Docker/Podman**:

### Container Requirements

Preprocessing containers should follow these conventions:

1. **Input Parameters**: Accept `--inputfile` and `--outputfolder` parameters
2. **File Processing**: Process the input file and generate outputs in the specified folder
3. **JSON Output**: Optionally output structured JSON on the last line for file tracking
4. **Metadata Support**: Include keyvalue pairs for annotation metadata

### Example Container Structure

See [ConvertLeica-Docker](https://github.com/Cellular-Imaging-Amsterdam-UMC/ConvertLeica-Docker) for a complete example.

```dockerfile
FROM python:3.9-slim

# Install your processing tools
RUN pip install your-processing-library

# Copy your processing script
COPY convert_script.py /app/
WORKDIR /app

# Entry point that accepts standard parameters
ENTRYPOINT ["python", "convert_script.py"]
```

Note: We suggest to keep the `user` in the Dockerfile as `ROOT` because non-root users might get into permission issues with the mounted I/O folders, especially on Windows. On Linux, we have the env option with `PODMAN_USERNS_MODE: keep-id` so that we can run also as non-root, but this doesn't work on Docker for Windows.

See the [security overview](./security_overview_privileged.md) for more details on the podman-in-podman or podman-in-docker setups, requirements, and issues.

### Podman Configuration

The system runs containers using Podman with these settings:

```yaml
# In docker-compose.yml
biomero-importer:
  privileged: true
  devices:
    - "/dev/fuse:/dev/fuse"
  security_opt:
    - "label=disable"
  environment:
    PODMAN_USERNS_MODE: keep-id  # For Linux user namespace mapping
```

### Preprocessing Parameters

Configure preprocessing in your database order:

```python
preprocessing = Preprocessing(
    container="cellularimagingcf/converter:latest",
    input_file="{Files}",  # Replaced by BIOMERO.importer with actual file path
    output_folder="/data",  # Mount point in container
    alt_output_folder="/out",  # Alternative output location
    extra_params={
        "saveoption": "single",
        "format": "tiff",
        "compression": "lzw"
    }
)
```

### JSON Output Format

For advanced file tracking, containers can output JSON on the last line:

```json
[
  {
    "name": "Image Name",   
    "full_path": "File Path relative to the docker data volume (i.e. inputfile path)",
    "alt_path": "/out/processed_image.tif",
    "keyvalues": [
      {"processing_method": "conversion"},
      {"original_format": "lsm"},
      {"compression": "lzw"}
    ]
  }
]
```

## Running the System

The BIOMERO.importer system is designed to run as a containerized service within the BIOMERO 2.0 ecosystem:

```bash
# Start the service (typically via docker-compose)
docker-compose up biomero-importer

# Check logs
docker-compose logs -f biomero-importer
```

## Monitoring and Debugging

### Log Files

The system generates several log files in `/auto-importer/logs/`:

- `app.logs`: Main application logs with all system activity
- `cli.<UUID>.logs`: OMERO CLI import logs for each upload order
- `cli.<UUID>.errs`: OMERO CLI error logs for each upload order

### Database Queries

Check system status with direct database queries:

```sql
-- View recent orders
SELECT uuid, stage, group_name, user_name, timestamp 
FROM imports 
ORDER BY timestamp DESC LIMIT 10;

-- Check pending orders
SELECT * FROM imports 
WHERE stage = 'Import Pending';

-- View preprocessing jobs
SELECT it.uuid, p.container, p.extra_params 
FROM imports it
JOIN imports_preprocessing p ON it.preprocessing_id = p.id
WHERE it.stage = 'Import Started';
```

### Testing the System

Use the system check script to verify setup:

```bash
# Inside the container
python tests/system_check.py
```

This creates a test upload order and verifies the complete ingestion pipeline.

## Error Handling

The system includes comprehensive error handling:

- **Dangling Orders**: Automatically marks stale orders as failed on startup
- **Retry Logic**: Database operations include retry mechanisms
- **Detailed Logging**: All operations are logged with appropriate detail levels
- **Graceful Shutdown**: Proper cleanup of resources and connections

## Integration with BIOMERO

The BIOMERO.importer system is designed to work seamlessly with the BIOMERO 2.0 environment:

- Shares the same PostgreSQL database (BIOMERO.db) for order coordination with BIOMERO.analyzer
- Integrates with BIOMERO's OMERO.biomero web plugin for a unified interface
- Provides audit trails for FAIR provenance

## Future Development

The current implementation is focused on:

1. **Enhanced Preprocessing**: Expanding containerized workflow support
2. **Performance Optimization**: Improved database polling and processing efficiency  
3. **Advanced Monitoring**: Better observability and alerting capabilities
4. **Multi-tenant Support**: Enhanced isolation and resource management

---

**Note**: This system replaces the previous file-based upload order approach. All order management is now database-driven using PostgreSQL (BIOMERO.db) and SQLAlchemy for improved reliability, scalability, and integration with BIOMERO.

## Data Access Architecture

The BIOMERO.importer system requires a shared storage architecture where data is accessible from multiple containers with read/write permissions. This is essential for in-place imports and preprocessing workflows.

### Storage Requirements

The system requires a **shared storage volume** (typically a Samba/CIFS mount or NFS) that is mounted identically across all containers:

- **OMERO Server**: For in-place imports using `ln_s` transfers
- **OMERO Web**: For OMERO.biomero plugin to browse and select files
- **BIOMERO.importer**: For reading source files and writing processed data
- **OMERO Workers**: For script access to data files

**Critical requirement**: All mounts must have **read/write (R/W) permissions**, not read-only.

### Mount Configuration

```yaml
# Example docker-compose.yml mounts
services:
  omeroserver:
    volumes:
      - "omero:/OMERO"
      - "./web/L-Drive:/data"  # Shared storage mounted as /data
      
  omeroweb:
    volumes:
      - "./web/L-Drive:/data:rw"  # Same mount path, R/W access
      
  biomero-importer:
    volumes:
      - "omero:/OMERO"
      - "./web/L-Drive:/data"  # Identical mount path for in-place imports
```

### In-Place Import Workflow

Image data remains on shared storage. Conventional imports use `transfer=ln_s`
to create filesystem symlinks. Zarr registration with `USE_REGISTER_ZARR=true`
instead records the physical Zarr location in `ExternalInfo.lsid` for the Zarr
pixel buffer. Preprocessing creates new outputs on shared storage. See
[Configuring the processed data folder](#configuring-the-processed-data-folder)
for the references that must remain valid when changing storage configuration.

### Preprocessing Data Flow

For conventional imports with preprocessing, the system follows this data flow:

```
Original Data (Remote Storage)
    ↓
Container Processing (On OMERO Server)
    ↓
Processed Data → Two Destinations:
    1. Remote Storage (PROCESSED_DATA_FOLDER subfolder; default .processed)
    2. Temporary Local Storage (alt_path)
    ↓
OMERO Import (from temporary storage)
    ↓
Symlink Redirect (to remote storage)
    ↓
Cleanup (temporary storage deleted)
```

For Zarr registration, `upload_files()` uses the converter's mapped `full_path`
on shared storage directly; it bypasses the temporary-storage import and
ManagedRepository symlink redirection shown above.

#### Why This Architecture?

1. **Performance**: Import from local temporary storage is faster than remote storage
2. **Reliability**: Avoid network issues during import process
3. **Storage Efficiency**: Final data resides on remote storage, not OMERO server
4. **Backup**: Processed data is preserved on remote storage

#### Implementation Details

In [`importer.py`](biomero_importer/utils/importer.py),
`DataProcessor.get_preprocessing_args()` creates the configured subfolder next
to the input and passes the matching converter path as `--outputfolder`.
`DataProcessor.run()` maps the converter's reported `full_path` and `alt_path`
back to host paths using the Podman mounts. `upload_files()` uses those reported
paths for registration or symlink redirection; it does not reconstruct the final
pixel location from a hardcoded `.processed` name.

### Metadata Integration

The system supports metadata inclusion through two mechanisms:

#### 1. CSV Metadata Files

Place a `metadata.csv` file alongside your import data:

```csv
key,value
acquisition_date,2024-01-15
magnification,63x
staining_method,DAPI
```

The system automatically detects and processes CSV files in:
- Original data directory
- Configured processed data directory (`PROCESSED_DATA_FOLDER`, default `.processed`)

#### 2. JSON Metadata from Preprocessing

Preprocessing containers can output metadata in their JSON response:

```json
[
  {
    "alt_path": "/out/processed_image.tif",
    "keyvalues": [
      {"processing_method": "deconvolution"},
      {"algorithm": "Richardson-Lucy"},
      {"iterations": "10"}
    ]
  }
]
```

This allows containers to:
- **Enrich metadata** by calling external APIs
- **Add processing parameters** automatically
- **Create metadata-only containers** that don't modify source data

### Example Deployment

#### Docker Compose Setup

```yaml
volumes:
  - "/mnt/shared-storage:/data"  # Shared storage mount
  - "omero:/OMERO"               # OMERO managed repository
```

#### Podman Setup (Linux)

```bash
podman run -d --rm --name biomero-importer \
    --privileged \
    --device /dev/fuse \
    --security-opt label=disable \
    -e OMERO_HOST=omeroserver \
    -e OMERO_USER=root \
    -e OMERO_PASSWORD=secret \
    -e OMERO_PORT=4064 \
    -e PODMAN_USERNS_MODE=keep-id \
    --network omero \
    --volume /mnt/datadisk/omero:/OMERO \
    --volume /mnt/L-Drive/basic/divg:/data \
    --volume "$(pwd)/logs/biomero-importer:/auto-importer/logs:Z" \
    --volume "$(pwd)/config:/auto-importer/config" \
    --userns=keep-id:uid=1000,gid=1000 \
    cellularimagingcf/biomero-importer:latest
```

### Storage Permissions

Ensure proper permissions on your shared storage.

Basic examples:

```bash
# Example for Linux hosts
sudo chmod -R 755 /mnt/shared-storage
sudo chown -R 1000:1000 /mnt/shared-storage

# For Samba/CIFS mounts, ensure the mount options allow R/W:
mount -t cifs //server/share /mnt/shared-storage -o username=user,rw,file_mode=0755,dir_mode=0755
```


### Troubleshooting Storage Issues

Common storage-related problems:

1. **Permission Denied**: Check R/W permissions on shared storage
2. **Import Failures**: Verify identical mount paths across all containers
3. **Symlink Errors**: Ensure OMERO managed repository is accessible
4. **Preprocessing Failures**: Check temporary storage space and permissions

Use these commands to diagnose:

```bash
# Check mount points
docker exec biomero-importer df -h

# Test file access
docker exec biomero-importer ls -la /data
docker exec biomero-importer touch /data/test-write-permissions

# Verify OMERO storage
docker exec biomero-importer ls -la /OMERO/ManagedRepository
```

This architecture ensures efficient, reliable data import while maintaining data integrity and providing flexibility for preprocessing workflows.

## BIOMERO 2.0 Integration

BIOMERO.importer is a core component of the BIOMERO 2.0 ecosystem, working alongside:

- **BIOMERO.analyzer**: For HPC-based image analysis workflows
- **BIOMERO.scripts**: For OMERO script-based workflow execution  
- **BIOMERO.db**: Shared PostgreSQL database for workflow coordination
- **OMERO.biomero**: Modern web interface for data import and analysis

Together, these components provide a comprehensive FAIR imaging platform for automated data management and analysis.

## Developer guide: schema changes and migrations

This project uses Alembic to manage database schema changes for BIOMERO.importer's tables only. Migrations run automatically on container startup (guarded by a Postgres advisory lock) and are isolated via a per-project version table `alembic_version_omeroadi`.

Below is a practical, copy-paste friendly guide to make and apply a schema change.

### 1) Create a virtual environment (Windows/Linux/macOS)

Windows PowerShell (Python 3.12, no activation required):

```powershell
py -3.12 -m venv .venv
.\.venv\Scripts\python -m pip install --upgrade pip
# Install Ice binaries first (follow the blog instructions for your OS)
# https://www.glencoesoftware.com/blog/2023/12/08/ice-binaries-for-omero.html
.\.venv\Scripts\python -m pip install -e .
```

Linux/macOS (Python 3.12):

```bash
python3.12 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
# Install Ice binaries first (follow the blog instructions for your OS)
# https://www.glencoesoftware.com/blog/2023/12/08/ice-binaries-for-omero.html
pip install -e .
```

Notes
- Editable install (-e) ensures your local package, including migrations, is importable.
- On Windows you can always prefix commands with .\.venv\Scripts\python -m ... instead of activating.

### 2) Ensure DB access for Alembic

Alembic autogenerate compares Models vs the live DB, so it must reach the database used by BIOMERO.importer.

Set the connection string as an environment variable:

```powershell
$env:INGEST_TRACKING_DB_URL = "postgresql://user:password@host:port/database"
```

If you are using the dev docker-compose, the Postgres service is typically exposed on a host port (e.g., 55432). Example:

```powershell
$env:INGEST_TRACKING_DB_URL = "postgresql://postgres:postgres@localhost:55432/biomero"
```

### 3) Make your SQLAlchemy model change

Edit the models in `biomero_importer/utils/ingest_tracker.py`. Keep changes minimal and run linters/tests as needed.

### 4) Generate a migration

The Alembic config is embedded under `biomero_importer/migrations/` and reads the URL from `INGEST_TRACKING_DB_URL`. Use python -m to avoid path issues.

```powershell
.\.venv\Scripts\python -m alembic -c biomero_importer\migrations\alembic.ini revision --autogenerate -m "your concise message"
```

Tips
- If Alembic reports “Target database is not up to date”, upgrade first (see next step) and re-run autogenerate.
- If you are adopting Alembic on an existing DB for the first time, see the optional “stamp” step below.

### 5) Apply migrations to your DB

Rebuild and restart your BIOMERO.importer container. The container will apply migrations automatically on startup when `ADI_RUN_MIGRATIONS=1` (default). This is handled by `biomero_importer/db_migrate.py` and uses a Postgres advisory lock to avoid races.

### 6) Commit the migration files

Add the new file(s) under `biomero_importer/migrations/versions/` to source control. These are included in the package so other environments (and the container) can run them.

### Optional: First-time adoption (stamp)

If your DB already has the BIOMERO.importer tables at the desired schema but no version table yet, you can baseline with a stamp so Alembic doesn't try to recreate history.

Two options:
1. Temporarily set `ADI_ALLOW_AUTO_STAMP=1` in the BIOMERO.importer container environment and restart the service once. The startup migration runner will stamp to head and then upgrade.
2. Or, run manually:
  ```powershell
  .\.venv\Scripts\python -m alembic -c biomero_importer\migrations\alembic.ini stamp head
  ```

After stamping, remove/disable the auto-stamp flag. Normal revisions and upgrades should be used going forward.

### How Alembic is scoped here

- Only BIOMERO.importer's tables are included via Alembic's `env.py` `include_object` filter. This prevents changes to other apps' tables in the same database.
- A dedicated version table `alembic_version_omeroadi` isolates BIOMERO.importer's migration history.

### Common issues

- autogenerate finds nothing: Ensure your model changes are in the BIOMERO.importer `Base` metadata and your `INGEST_TRACKING_DB_URL` points to the correct DB.
- autogenerate complains DB not up to date: Run upgrade head, then re-run autogenerate.
- Missing template `script.py.mako`: It's included under `biomero_importer/migrations/`; ensure your editable install points to your working tree.

### Quick reference

```powershell
# Set DB URL for alembic
$env:INGEST_TRACKING_DB_URL = "postgresql://postgres:postgres@localhost:55432/biomero"

# Create venv and install
py -3.12 -m venv .venv
# Install Ice binaries first (follow the blog instructions for your OS)
# https://www.glencoesoftware.com/blog/2023/12/08/ice-binaries-for-omero.html
.\.venv\Scripts\python -m pip install -e .

# Generate and apply migration
.\.venv\Scripts\python -m alembic -c biomero_importer\migrations\alembic.ini revision --autogenerate -m "add new column"
.\.venv\Scripts\python -m alembic -c biomero_importer\migrations\alembic.ini upgrade head
```

## LICENSE

License changed to GPL-2.0 (starting version 1.3), as this work depends on `omero-py` and `ezomero` libraries for the OMERO import and session management.
