Metadata-Version: 2.4
Name: gandalf-csr
Version: 2.0.0
Summary: Fast path finding in large knowledge graphs
Home-page: https://github.com/ranking-agent/gandalf
Author: Max Wang
Author-email: Max Wang <max@covar.com>
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: bmt>=1.4.8
Requires-Dist: lmdb>=1.4.0
Requires-Dist: msgpack>=1.0.0
Requires-Dist: numpy>=1.20.0
Requires-Dist: translator_tom>=2.1.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0; extra == "dev"
Requires-Dist: black>=22.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0.12; extra == "dev"
Provides-Extra: server
Requires-Dist: fastapi>=0.100.0; extra == "server"
Requires-Dist: httpx>=0.24.0; extra == "server"
Requires-Dist: orjson>=3.9.0; extra == "server"
Requires-Dist: psutil>=5.9.0; extra == "server"
Requires-Dist: pydantic>=2.0.0; extra == "server"
Requires-Dist: pydantic-settings>=2.12.0; extra == "server"
Requires-Dist: uvicorn>=0.20.0; extra == "server"
Requires-Dist: zstandard>=0.22.0; extra == "server"
Provides-Extra: mongo
Requires-Dist: pymongo>=4.0; extra == "mongo"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# GANDALF

Graph Analysis Navigator for Discovery And Link Finding

A high-performance Python library and [Translator](https://ncats.nih.gov/translator)-compatible TRAPI server for fast path finding in large biomedical knowledge graphs.

## Features

- **Compressed Sparse Row (CSR)** graph representation for memory-efficient storage of 10M+ node, 38M+ edge graphs
- **Bidirectional search** for optimal path-finding performance
- **O(1) property lookups** via hash indexing
- **Predicate filtering** to reduce path explosion
- **Qualifier filtering** for advanced edge constraints (aspect, direction, mechanism)
- **Attribute constraints** on edges and nodes, including filtering edges by specific PubMed IDs
- **Subclass expansion** via Biolink Model Toolkit with configurable depth
- **Batch property enrichment** — enrich only final paths, not intermediate results
- **Diagnostic tools** to understand path counts and explosion
- **TRAPI 2.0 compatible** REST API with Plater-compatible endpoints, modelled
  with [`translator_tom`](https://github.com/NCATSTranslator/TRAPIObjectModeling)
- **Async query support** with callback URLs
- **Dehydrated mode** for lightweight responses that skip edge and node attribute enrichment
- **OpenTelemetry tracing** with Jaeger integration

## Installation

**Recommended: Use a virtual environment**

Some transitive dependencies (e.g., `stringcase`, `pytest-logging`) require modern pip/setuptools to build correctly. Using a virtual environment ensures you have updated tools.

```bash
# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Upgrade pip and setuptools (important for building dependencies)
pip install --upgrade pip setuptools wheel

# Install the core package
pip install -e .

# Install with server dependencies (FastAPI, uvicorn, etc.)
pip install -e ".[server]"

# Install with dev dependencies (pytest, black, mypy)
pip install -e ".[dev]"
```

## Quick Start

### Unzipping a full Translator KGX

```bash
tar -xvf translator_kg.tar.zst
```

This will output a `nodes.jsonl` and `edges.jsonl` file.

### Build a graph from JSONL

```python
from gandalf import build_graph_from_jsonl

# Build with ontology filtering
graph = build_graph_from_jsonl(
    edges_path="data/raw/edges.jsonl",
    nodes_path="data/raw/nodes.jsonl",
)

# Save for fast loading
graph.save_mmap("data/processed/gandalf_mmap")
```

> **Upgrading to TRAPI 2.0 requires a rebuild.** Some of what 2.0 mandates is
> baked into the serialized graph, so a graph built by an earlier version
> cannot serve a conformant response. `CSRGraph.load_mmap` refuses such a
> graph outright with a `GraphFormatError` naming the rebuild, rather than
> starting up and serving edges with no `knowledge_level` / `agent_type`.
>
> Baked in at build time (a rebuild is the only way to change these):
> `knowledge_level` / `agent_type` on every edge, `Node.name` omitted when
> unknown, `Node.categories` defaulted to `biolink:NamedThing`,
> `RetrievalSource.upstream_resource_ids` omitted when empty, and the
> persisted `meta_kg.json` / `sri_testing_data.json`.
>
> Everything else 2.0 changed is computed per query and takes effect on
> deploy: binding shapes, `QEdge.constraints`, the `parameters` object and its
> timeout, the response envelope, and the remaining null / empty-container
> rules.

### Query paths (TRAPI format)

```python
from gandalf import CSRGraph, lookup

# Load graph (takes ~1-2 seconds)
graph = CSRGraph.load_mmap("data/processed/gandalf_mmap")

# Execute a TRAPI query
response = lookup(
    graph,
    {
        "message": {
            "query_graph": {
                "nodes": {
                    "n0": {"ids": ["CHEBI:45783"]},
                    "n1": {"categories": ["biolink:Gene"]},
                    "n2": {"categories": ["biolink:Disease"]}
                },
                "edges": {
                    "e0": {"subject": "n0", "object": "n1", "predicates": ["biolink:affects"]},
                    "e1": {"subject": "n1", "object": "n2"}
                }
            }
        }
    },
    subclass=True,
    subclass_depth=1,
)

print(f"Found {len(response['message']['results'])} paths")
```

### Constraining query edges

TRAPI 2.0 gathers every constraint on a query edge into one `constraints`
object. All constraints given must hold:

```python
"edges": {
    "e0": {
        "subject": "n0",
        "object": "n1",
        "predicates": ["biolink:affects"],
        "constraints": {
            "knowledge_level": {
                "behavior": "ALLOW",
                "values": ["knowledge_assertion"],
            },
            "agent_type": {"behavior": "DENY", "values": ["text_mining_agent"]},
            "sources": {
                "behavior": "ALLOW",
                "values": ["infores:ctd"],
                "primary_only": True,
            },
            "qualifiers": [
                {"biolink:object_aspect_qualifier": "activity"},
            ],
            "attributes": [
                {"id": "biolink:publications", "operator": "==", "value": [...]},
            ],
        },
    }
}
```

- `knowledge_level` / `agent_type` — allow or deny Biolink values on the bound
  edges. `ALLOW` needs at least one listed value to be present; `DENY` needs
  none of them to be.
- `sources` — the same allow/deny over the infores CURIEs in an edge's
  `sources`. `primary_only` narrows the check to the source whose role is
  `primary_knowledge_source`, so a constraint can ignore aggregators.
- `qualifiers` — a list of qualifier mappings. AND within one mapping, OR
  between them. Values expand through the Biolink hierarchy, so a query for a
  parent value also matches edges carrying a child value.
- `attributes` — attribute constraints, evaluated against the edge's
  attributes. Query nodes accept the same list directly under `constraints`.
  `"not": true` negates one.

Migrating from TRAPI 1.x: `qualifier_constraints` is now
`constraints.qualifiers` (and its `qualifier_set` list of
`qualifier_type_id`/`qualifier_value` pairs collapses into a single mapping),
and `attribute_constraints` is now `constraints.attributes`. Gandalf rejects
the old field names with a 400 rather than ignoring them, so a stale client
never silently receives unfiltered results.

#### Filtering edges by PubMed ID

Attribute values are often lists — `publications` above all — and every
operator except `===` is applied to each member, so `==` reads as "contains".
Filtering an edge down to specific PubMed IDs is therefore plain equality:

```python
"constraints": {
    "attributes": [
        {
            "id": "biolink:publications",
            "operator": "==",
            "value": ["PMID:23456789", "PMID:11111111"],
        }
    ]
}
```

Only edges citing at least one of those PMIDs survive. A list `value` means
"any of"; a single string constrains to one publication. Publication
identifiers are compared canonically, so `PMID:23456789`, `pubmed:23456789`,
`https://pubmed.ncbi.nlm.nih.gov/23456789` and the bare `23456789` all select
the same article — unlike the `matches` operator, which does a substring
regex and would also accept `PMID:234567890`.

## Architecture

The package uses a three-stage pipeline:

1. **Topology Search** (fast) - Find all paths using indices only
2. **Filtering** (medium) - Apply business logic on necessary node or edge properties
3. **Enrichment** (batch) - Load all properties for final paths only

This separation allows filtering millions of paths before expensive property lookups.

## REST API

The server exposes Plater-compatible TRAPI endpoints on port 6429.

**Run the development server:**

```bash
python gandalf/main.py
```

**Run the production server:**

```bash
gunicorn gandalf.server:APP -c gunicorn.conf.py
```

### Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/` | Redirect to `/docs` |
| `GET` | `/docs` | Swagger UI documentation |
| `GET` | `/metadata` | Graph statistics and metadata |
| `GET` | `/node_degree/{curie}` | Total degree (in + out) of a node |
| `GET` | `/meta_knowledge_graph` | Meta KG with predicates, categories, and counts |
| `GET` | `/sri_testing_data` | Representative edges for SRI Testing Harness |
| `POST` | `/query` | Synchronous TRAPI query |
| `POST` | `/asyncquery` | Async TRAPI query with callback URL |

Both `/query` and `/asyncquery` accept a single optional query parameter:
- `?profile=true` — Emit per-stage timing diagnostics into `message.logs`

All other request configuration lives under the body's `parameters` object,
which TRAPI 2.0 defines for query-time settings that do not change what a query
means. The server repeats it back in the response, as the spec requires.

```json
{
  "message": { "query_graph": { ... } },
  "parameters": {
    "timeout": 60,
    "log_level": "INFO",
    "bypass_cache": false,
    "subclass": true,
    "subclass_depth": 1,
    "dehydrated": false,
    "filter_config": { "max_node_degree": 50 },
    "annotator_config": {}
  }
}
```

Responses never serialize a null, and never serialize an empty container for a
property whose schema forbids one (`Edge.qualifiers`,
`RetrievalSource.upstream_resource_ids`, `message.auxiliary_graphs`, `logs`, …)
— TRAPI 2.0 is OpenAPI 3.1 and dropped `nullable`, so an absent value is an
absent property. `message.results` is deliberately still `[]` when a query
matched nothing, which is what 2.0 asks for.

TRAPI's own parameters:

- `timeout` (number): Seconds the client is willing to wait. When the budget is
  spent the query stops and the response carries a `Timeout` status with the
  logs from the work done. A negative value disables the server's default
  timeout (`GANDALF_QUERY_TIMEOUT`). A value below `GANDALF_MIN_QUERY_TIMEOUT`
  is refused up front with HTTP 409, since the server knows it cannot answer
  that fast.
- `log_level` (string): Least critical level of logs to return — `ERROR`,
  `WARNING`, `INFO` or `DEBUG`. (This moved here from the request's top level
  in TRAPI 2.0.)
- `bypass_cache` (bool): Accepted and has no effect — gandalf answers from its
  own graph and holds no query cache.

Gandalf's own parameters:

- `subclass` (bool): Enable biolink subclass inference (default `true`)
- `subclass_depth` (int): Maximum `subclass_of` hops (default `1`)
- `dehydrated` (bool): Return the smallest useful response — edges carry only
  subject, object, predicate, `knowledge_level` and `agent_type`, with no
  attributes and no `sources` (auto-enabled for very large result sets).
  TRAPI 2.0 requires `sources`, so a dehydrated response is deliberately not
  schema-valid: the mode trades conformance for size, and rehydrating one
  (see `rehydrate`) restores a conformant response
- `rehydrate` (bool): When true, the server skips the graph lookup and **only** enriches the `knowledge_graph` already supplied in `message` — used to re-enrich a previously dehydrated response
- `filter_config` (object): Plugin-defined node filter settings (each NodeFilter plugin reads its own key)
- `annotator_config` (object): Per-request opt-in response-annotator settings (each key activates one annotator plugin)

## CLI Commands

```bash
# Build a CSR graph from JSONL node/edge files
gandalf-build --edges data/edges.jsonl --nodes data/nodes.jsonl --output data/graph_mmap/

# Query paths from the command line
gandalf-query --graph data/graph_mmap/ --start "CHEBI:45783" --end "MONDO:0004979"

# Diagnose path explosion between two nodes
gandalf-diagnose --graph data/graph_mmap/ --start "CHEBI:45783" --end "MONDO:0004979"
```

## Configuration

The server is configured via environment variables (prefixed with `GANDALF_`):

### Core

| Variable | Default | Description |
|----------|---------|-------------|
| `GANDALF_GRAPH_PATH` | `/data/graph` | Path to the mmap graph directory |
| `GANDALF_GRAPH_FORMAT` | `auto` | Graph format (`auto` or `mmap`) |
| `GANDALF_LOAD_MMAPS_INTO_MEMORY` | `false` | Load memory-mapped arrays fully into RAM |
| `GANDALF_LOG_LEVEL` | `INFO` | Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
| `GANDALF_LOG_FORMAT` | `text` | Log format (`text` for human-readable, `json` for structured) |
| `GANDALF_CORS_ORIGINS` | `*` | Comma-separated list of allowed CORS origins |
| `GANDALF_MAX_REQUEST_SIZE_MB` | `10` | Maximum request body size in MB |
| `GANDALF_RATE_LIMIT` | `0` | Max requests per minute per client IP (0 = disabled) |
| `GANDALF_SKIP_PRELOAD` | `false` | Skip module-level graph loading |
| `GANDALF_WORKERS` | `2` | Gunicorn worker count |

### Search Tuning

| Variable | Default | Description |
|----------|---------|-------------|
| `GANDALF_LARGE_RESULT_THRESHOLD` | `50000` | Path count threshold for auto-dehydrated responses |
| `GANDALF_MAX_PATH_LIMIT` | `0` | Max intermediate paths during joins (0 = unlimited) |
| `GANDALF_DEBUG_PATHS_TSV` | _(empty)_ | File path to write debug TSV of reconstructed paths |

### TRAPI

| Variable | Default | Description |
|----------|---------|-------------|
| `GANDALF_QUERY_TIMEOUT` | `0` | Server default for `parameters.timeout`, in seconds (0 = no timeout) |
| `GANDALF_MIN_QUERY_TIMEOUT` | `1.0` | Shortest `parameters.timeout` the server accepts; below this it answers HTTP 409 |
| `GANDALF_DATA_RELEASE_VERSIONS` | _(empty)_ | JSON object of source data versions reported as `Response.data_release_versions`, e.g. `{"translator_kg": "2026_06_21"}` |
| `GANDALF_BIOLINK_VERSION` | `4.3.2` | Biolink Model version reported and used for predicate/qualifier expansion |

### Server Identity

| Variable | Default | Description |
|----------|---------|-------------|
| `GANDALF_SERVER_URL` | `http://localhost:6429` | Public URL of this instance |
| `GANDALF_SERVER_MATURITY` | `development` | Maturity level for TRAPI metadata |
| `GANDALF_SERVER_LOCATION` | `RENCI` | Server location for TRAPI metadata |
| `GANDALF_INFORES` | `infores:gandalf` | Translator infores identifier |

### Automat Heartbeat

| Variable | Default | Description |
|----------|---------|-------------|
| `GANDALF_AUTOMAT_HOST` | _(empty, disabled)_ | Automat cluster URL for registration |
| `GANDALF_HEART_RATE` | `30` | Seconds between heartbeats |
| `GANDALF_SERVICE_ADDRESS` | _(empty)_ | Reachable address of this instance |
| `GANDALF_WEB_PORT` | `8080` | Port for heartbeat registration |

### Observability

| Variable | Default | Description |
|----------|---------|-------------|
| `GANDALF_OTEL_ENABLED` | `true` | Enable OpenTelemetry tracing |
| `GANDALF_OTEL_SERVICE_NAME` | `gandalf` | Service name for traces |
| `GANDALF_JAEGER_HOST` | `http://jaeger` | Jaeger collector host |
| `GANDALF_JAEGER_PORT` | `4317` | Jaeger collector gRPC port |

## Docker

```bash
# Build the image
docker build -t gandalf .

# Run with a graph volume
docker run -p 6429:6429 \
  -v /path/to/graph:/data/graph \
  -e GANDALF_GRAPH_PATH=/data/graph \
  gandalf
```

## Verifying the Server

```bash
# Check graph metadata
curl http://localhost:6429/metadata

# Browse the API docs
open http://localhost:6429/docs
```

## Releases
- Make a release in GitHub to run a GitHub Action that pushes a gandalf to ghcr
- Run this on the mmap folder: `tar -czvf gandalf_mmap_<date>.tar.gz gandalf_mmap`
- Upload the tar.gz file to a public file server
- Update any helm charts and deploy
