Metadata-Version: 2.4
Name: icat_indexer
Version: 0.1.0
Summary: index in ES ICAT data
Author-email: ESRF <marjolaine.bodin@esrf.fr>
License: # MIT License
        
        **Copyright (c) 2026 European Synchrotron Radiation Facility**
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of
        this software and associated documentation files (the "Software"), to deal in
        the Software without restriction, including without limitation the rights to
        use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
        the Software, and to permit persons to whom the Software is furnished to do so,
        subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
        FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
        COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
        IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
        CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
        
Project-URL: Homepage, https://gitlab.esrf.fr/icat/data-sync/icat-indexer
Project-URL: Documentation, https://icat_indexer.readthedocs.io/
Project-URL: Repository, https://gitlab.esrf.fr/icat/data-sync/icat-indexer/
Project-URL: Issues, https://gitlab.esrf.fr/icat/data-sync/icat-indexer/issues
Project-URL: Changelog, https://gitlab.esrf.fr/icat/data-sync/icat-indexer/-/blob/main/CHANGELOG.md
Keywords: icat,elastic search,metadata
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: pyicat-plus
Requires-Dist: icat_esrf_definitions
Requires-Dist: elasticsearch
Requires-Dist: apscheduler>=3.10
Requires-Dist: stomp.py
Requires-Dist: pydantic-settings
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Provides-Extra: dev
Requires-Dist: icat_indexer[test]; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Provides-Extra: doc
Requires-Dist: icat_indexer[test]; extra == "doc"
Requires-Dist: sphinx>=4.5; extra == "doc"
Requires-Dist: sphinx-autodoc-typehints>=1.16; extra == "doc"
Requires-Dist: pydata-sphinx-theme; extra == "doc"
Requires-Dist: sphinx-copybutton; extra == "doc"
Dynamic: license-file

# icat_indexer

Synchronises ICAT dataset metadata into Elasticsearch.

Two complementary sync mechanisms run in parallel:

- **Nightly scheduler**: indexes all investigations whose release date falls on today
- **ActiveMQ listener**: reacts in real time when a release date changes or a dataset DOI is minted

When an investigation goes back under embargo (release date moved to the future), its datasets are automatically removed from the index.

The ES index mapping is generated automatically from `icat_esrf_definitions` models, so it stays in sync with the metadata schema without manual maintenance.


## Installation

```bash
pip install icat_indexer
```

Or in development mode:

```bash
pip install -e ".[dev]"
```

## Configuration

Copy `.env.example` to `.env` and fill in your credentials:

```bash
cp .env.example .env
```

| Variable | Description | Default |
|---|---|---|
| `ICAT_URL` | ICAT+ base URL | — |
| `ICAT_USERNAME` | ICAT+ username | — |
| `ICAT_PASSWORD` | ICAT+ password | — |
| `ICAT_AUTH_PLUGIN` | Auth plugin | `db` |
| `ES_URL` | Elasticsearch URL | — |
| `ES_INDEX` | Index name | `public_datasets` |
| `ES_USERNAME` | ES username | — |
| `ES_PASSWORD` | ES password | — |
| `ACTIVEMQ_HOST` | ActiveMQ host | — |
| `ACTIVEMQ_PORT` | ActiveMQ STOMP port | `61613` |
| `ACTIVEMQ_QUEUE` | Queue to subscribe to | `icatPublicationEvents` |
| `SCHEDULER_HOUR` | Nightly sync hour (UTC) | `1` |
| `SCHEDULER_MINUTE` | Nightly sync minute (UTC) | `0` |

## Usage

Run the nightly scheduler:

```bash
icat-indexer-scheduler
```

Run the ActiveMQ listener:

```bash
icat-indexer-listener
```

Both can run simultaneously as separate processes or services.

## ActiveMQ message format

**Investigation release date changed** (index or delete all datasets):
```json
{"type": "investigation_release_changed", "investigationId": "12345"}
```

**Dataset DOI minted** (always index, never delete):
```json
{"type": "dataset_doi_minted", "datasetIds": ["67890", "75315"]}
```

## Development

```bash
pip install -e ".[dev]"
pytest
ruff check . --fix  
ruff format  
```

## License

MIT - see [LICENSE.md](LICENSE.md).

## Documentation

https://icat_indexer.readthedocs.io/
