Metadata-Version: 2.4
Name: open-kgo
Version: 0.3.1
Summary: Open Knowledge Graphs and Ontologies plugin for mloda.
Author-email: Tom Kaltofen <tomkaltofen@mloda.ai>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/mloda-ai/open-kgo
Project-URL: Repository, https://github.com/mloda-ai/open-kgo
Project-URL: Issues, https://github.com/mloda-ai/open-kgo/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mloda>=0.10.0
Provides-Extra: dev
Requires-Dist: tox; extra == "dev"
Requires-Dist: tox-uv; extra == "dev"
Requires-Dist: uv>=0.11.6; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: bandit; extra == "dev"
Requires-Dist: mloda-testing>=0.3.2; extra == "dev"
Provides-Extra: kg-rdf
Requires-Dist: rdflib>=7.0; extra == "kg-rdf"
Requires-Dist: pyoxigraph>=0.5.9; extra == "kg-rdf"
Provides-Extra: kg-embedded
Requires-Dist: networkx>=3.2; extra == "kg-embedded"
Requires-Dist: igraph>=0.11; extra == "kg-embedded"
Provides-Extra: kg-network-pg
Requires-Dist: kuzu>=0.4; extra == "kg-network-pg"
Requires-Dist: networkx>=3.2; extra == "kg-network-pg"
Requires-Dist: grand-cypher>=0.12; extra == "kg-network-pg"
Provides-Extra: kg-memory
Requires-Dist: networkx>=3.2; extra == "kg-memory"
Provides-Extra: kg-ontology
Requires-Dist: pyyaml>=6.0; extra == "kg-ontology"
Provides-Extra: kg-semantic-field
Requires-Dist: pyyaml>=6.0; extra == "kg-semantic-field"
Provides-Extra: kg-rest
Provides-Extra: kg-lineage
Provides-Extra: kg-codebuild
Provides-Extra: kg-saas
Provides-Extra: kg-citation
Provides-Extra: kg-all
Requires-Dist: open-kgo[kg-citation,kg-codebuild,kg-embedded,kg-lineage,kg-memory,kg-network-pg,kg-ontology,kg-rdf,kg-rest,kg-saas,kg-semantic-field]; extra == "kg-all"
Provides-Extra: demo
Requires-Dist: marimo; extra == "demo"
Requires-Dist: open-kgo[kg-all]; extra == "demo"
Dynamic: license-file

[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![mloda](https://img.shields.io/badge/built%20with-mloda-blue.svg)](https://github.com/mloda-ai/mloda)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)

# open-kgo

Open Knowledge Graphs and Ontologies plugin for [mloda](https://github.com/mloda-ai/mloda): nine connector families covering the knowledge-graph landscape, from SPARQL endpoints to SBOMs to agent memory, all behind one declarative `Feature` interface. Every connector and demo runs offline against in-memory libraries or committed fixtures. No Docker, no network.

## At a glance

| Section | What you'll find |
|---|---|
| [Quickstart](#quickstart) | Run a SPARQL query against a shipped sample file in under a minute |
| [The nine connector families](#the-nine-connector-families) | The core of this repo: a 9-family KG connector taxonomy with two plugins each |
| [Semantic Fields — Layer 2](#semantic-fields--layer-2) | Continuous, schema-grounded entity scoring via DC circuit model |
| [DiscoveryEngine — Layer 3](#discoveryengine--layer-3) | Beam search over the EM field — ranked typed paths, no LLM calls required |
| [Demos](#demos) | Marimo notebooks and evaluation harnesses, all offline |
| [Data and acknowledgments](#data-and-acknowledgments) | Where the sample data comes from |
| [Development setup](#development-setup) | uv, tox, and the individual checks |
| [Related repositories and documentation](#related-repositories-and-documentation) | mloda core, the plugin registry, and development guides |

## Quickstart

Install the connectors from PyPI and run a SPARQL query against a small Turtle file:

```bash
pip install "open-kgo[kg-all]"
```

(Working from a clone instead? `uv sync --extra kg-all`, see [Development setup](#development-setup).)

```python
from pathlib import Path

from mloda.user import DataAccessCollection, Feature, Options, mloda

# Importing the plugin module registers the rdflib_sparql feature group.
import open_kgo.feature_groups.kg.rdf.rdflib_sparql  # noqa: F401
from open_kgo.feature_groups.kg.base import PythonDictFramework

# Point at any RDF file. Here: a three-triple sample written on the spot.
ttl = Path("sample.ttl")
ttl.write_text(
    "@prefix foaf: <http://xmlns.com/foaf/0.1/> .\n"
    "@prefix ex: <http://example.org/> .\n"
    "ex:alice foaf:knows ex:bob .\n"
    "ex:bob foaf:knows ex:carol .\n"
)

feature = Feature(
    "rdflib_sparql__knows",
    options=Options(context={
        "query_text": "PREFIX foaf: <http://xmlns.com/foaf/0.1/> "
                      "SELECT ?s ?o WHERE { ?s foaf:knows ?o }",
    }),
)

partitions = mloda.run_all(
    [feature],
    compute_frameworks={PythonDictFramework},
    data_access_collection=DataAccessCollection(
        credentials=[{"rdflib_sparql": {"locator": str(ttl), "result_limit": 100}}],
    ),
)

for partition in partitions:
    for row in partition.get(feature.name, []):
        print(row)
```

Swap `rdflib_sparql` for any of the nine connector families below: same `Feature` to `mloda.run_all` shape, different reader.

## The nine connector families

`open_kgo/feature_groups/kg/` ships a connector taxonomy derived from a 103-system survey. Each family is a shared reader and feature-group base plus **two** concrete plugins running against in-memory libraries or local file fixtures:

| Family | What it connects to | Concrete plugins |
|---|---|---|
| `network_pg` | Property-graph databases with a vendor query language (Neo4j, Memgraph, Neptune, ...) | `KuzuCypherReader`, `GrandCypherReader` |
| `rdf` | RDF triple stores queried with SPARQL | `RdfLibSparqlReader`, `OxigraphSparqlReader` |
| `embedded` | In-process graph libraries with no network endpoint | `NetworkxEmbeddedReader`, `IGraphEmbeddedReader` |
| `rest_public` | Public REST (non-SPARQL) KG APIs (OpenAlex, ConceptNet, STRING, ...) | `FileFixtureRestReader`, `FileFixturePagedRestReader` |
| `lineage` | Metadata and data-lineage graphs (dbt, OpenLineage, DataHub, ...) | `DbtManifestReader`, `OpenLineageReader` |
| `code_build` | Code, build, and SBOM dependency graphs (CycloneDX, SPDX, ...) | `CycloneDxSbomReader`, `SpdxSbomReader` |
| `saas_authz` | SaaS and authorization tuple stores (OpenFGA, SpiceDB, Microsoft Graph, ...) | `InProcessTupleStoreReader`, `PaginatedTupleStoreReader` |
| `agent_memory` | LLM agent memory and GraphRAG graphs (Letta, Zep, Mem0, ...) | `NetworkxMemoryReader`, `GraphWalkMemoryReader` |
| `citation_rest` | Citation and scientific REST APIs (Reactome, OpenAlex citations, ...) | `FileFixtureCitationReader`, `PaginatedCitationReader` |

See [`open_kgo/feature_groups/kg/README.md`](open_kgo/feature_groups/kg/README.md) for the full family map, the plugin anatomy, and what the prototype does and does not validate.

Install all KG extras with: `pip install "open-kgo[kg-all]"` (from a clone: `uv sync --extra kg-all`).

> **One feature per call.** KG readers dispatch a single feature per load: every
> reader rejects a multi-feature `FeatureSet` rather than silently labelling all
> rows with one feature name. Request features individually (one `Feature` per
> `mloda.run_all` slot) rather than batching `N` of them into a single reader call.

> **No-Docker testing policy.** Every connector test runs against rdflib, networkx, kuzu (embedded), or file fixtures. No Docker, no external services, no network calls.

## Semantic Fields — Layer 2

The connector families (Layer 1 validation) answer: *"is this traversal valid?"* — binary yes/no.

**SemanticField** (Layer 2) answers: *"how relevant is this entity to my query?"* — a continuous score, grounded in DC electrical circuit theory.

The knowledge graph is modelled as a resistor network. Ontology-declared relationship weights become conductances. A query is expressed as two anchor nodes — a **source** (high voltage) and a **sink** (low voltage, ground). The solver finds the electric potential at every entity via the conductance-weighted graph Laplacian, then scores each entity by the current it carries between source and sink.

**Why current?** Current only flows through entities that bridge *both* anchors. An entity connected to only one side floats to that side's extreme voltage — zero potential difference, zero current, automatically excluded. No relation-type filtering needed; the circuit topology enforces the AND constraint.

```python
from open_kgo.feature_groups.kg.ontology import SemanticField

# AND query: Sci-Fi films by Nolan
scores = SemanticField.compute_and(
    namespace="metaqa",
    edges=subgraph_edges,          # 2-hop neighbourhood of both anchors
    source={"Nolan": 1.0},         # high-voltage anchor
    sink={"Sci-Fi": 0.0},          # ground
)
# → {"Interstellar": 0.394, "Inception": 0.312, "Dark Knight": 0.0, ...}
```

Relationship weights are declared in the ontology YAML and flow directly into the solver as conductances:

```yaml
relationships:
  directed_by: { domain: Movie, range: Person, weight: 0.9 }
  has_genre:   { domain: Movie, range: Genre,  weight: 0.7 }
  has_tags:    { domain: Movie, range: Tag,    weight: 0.2 }
```

The solver is pure Python (no numpy) with an optional numba JIT kernel that gives a **70× speedup** at ~5 000-node subgraphs; it falls back silently when numba is absent.

| Layer | What it does | Status |
|---|---|---|
| L1 — OntologyRegistry | Binary valid/invalid edge checks, YAML-declared | Built |
| L2 — SemanticField | Continuous entity scoring via DC circuit model | Built |
| L3 — Discovery Engine | Guided multi-hop traversal by field strength | Built |

Install: `pip install "open-kgo[kg-semantic-field]"` (from a clone: `uv sync --extra kg-semantic-field`)

Full visual explainer: [`demo/semantic_field_explainer.html`](demo/semantic_field_explainer.html) — open in any browser, no server needed.

## DiscoveryEngine — Layer 3

**DiscoveryEngine** (Layer 3) answers: *"what is the best route through the graph from source to goal?"* — ranked typed paths, guided by the EM field gradient from Layer 2. No LLM calls, no oracle scores, no sampling.

Layer 2 already solved the hard problem: it computed a voltage at every entity. The edge current between adjacent nodes `i` and `j` is `G(i,j) × |V(i) - V(j)|`. That current **is the beam search heuristic** — edges that carry the most signal between source and sink are expanded first. Dead-end branches carry zero current and never enter the beam.

Path scores use **bottleneck current** — the minimum edge current along the path. A path scores high only when every hop is strongly conducting, making the score interpretable: it is the weakest link in the conducting chain.

```python
from open_kgo.feature_groups.kg.ontology import DiscoveryEngine

# Ranked paths: which movies connect Nolan to Sci-Fi?
paths = DiscoveryEngine.find_paths(
    namespace="movie",
    edges=subgraph_edges,
    source={"Christopher Nolan": 1.0},   # high-voltage anchor
    sink={"Science Fiction": 0.0},        # ground
    beam_width=5,
    max_depth=4,
)
# → [
#     DiscoveredPath(nodes=("Christopher Nolan", "Inception", "Science Fiction"),
#                    relations=("directed_by", "has_genre"), score=0.301),
#     ...
# ]

# Minimal current-carrying subgraph — the explanation circuit
circuit_edges = DiscoveryEngine.extract_circuit(
    namespace="movie",
    edges=subgraph_edges,
    source={"Christopher Nolan": 1.0},
    sink={"Science Fiction": 0.0},
    current_threshold=0.01,
)
# → 25 of 116 edges — dead-end actors and unrelated genres excluded
```

### Where it helps

**Without an LLM (pure graph analytics):**
- **Multi-hop path finding** — find all typed routes between two entities ranked by semantic strength, not just shortest hop count.
- **Influence analysis** — extract the minimal subgraph that bridges a source and a target. Prunes dead-end branches automatically (zero potential difference = zero current).
- **Bottleneck detection** — the edge with lowest current on the best path is the weakest semantic link. Flag it for curation or ontology-weight tuning.
- **Subgraph extraction** — `extract_circuit` returns a compact, query-specific subgraph in one pass. Cheaper and more targeted than community detection.

**With an LLM (context and loop engineering):**
- **Context engineering** — hand the LLM `extract_circuit` output rather than a flat bag of triples or a static GraphRAG community. The circuit is already filtered to the edges live for *this* query. Smaller context window, higher precision.
- **Loop engineering** — between agent iterations, re-solve the field with updated anchors and call `find_paths` again. The paths from one step become the anchor candidates for the next step. The field carries forward the accumulated relevance signal.
- **GraphRAG replacement** — GraphRAG pre-computes community summaries over the whole graph and serves the nearest community at query time. The EM field is re-solved per query: source and sink anchors parameterize the field, so every call gets a subgraph tuned to the specific question, not the nearest pre-computed partition.
- **Think-on-Graph improvement** — Think-on-Graph ranks candidate expansions by asking the LLM at each hop (one LLM call per edge explored). `find_paths` replaces that with the pre-computed edge current: zero LLM calls during traversal, same ranked result.

| Layer | What it does | Status |
|---|---|---|
| L1 — OntologyRegistry | Binary valid/invalid edge checks, YAML-declared | Built |
| L2 — SemanticField | Continuous entity scoring via DC circuit model | Built |
| L3 — DiscoveryEngine | Beam search over the EM field, ranked typed paths | Built |

Install: `uv sync --extra kg-semantic-field`

Interactive demo: `marimo edit demo/demo_discovery.py`

## Demos

Three marimo notebooks plus two evaluation harnesses live under `demo/`:

- `demo/demo_kg_connectors.py`: surface tour of all 9 families against the shipped fixtures.
- `demo/demo_kg_build_repo.py`: builds an RDF graph from this repo (filesystem `repo:contains` + Python `repo:imports`), serializes to Turtle, and runs five SPARQL queries through `RdfLibSparqlReader` via `mloda.run_all`.
- `demo/demo_kg_ontology.py`: walks the ontology layer end to end.
- `demo/demo_semantic_field.py`: SemanticField Layer 2 — interactive director + genre query against MetaQA, with live scoring.
- `demo/demo_discovery.py`: DiscoveryEngine Layer 3 — beam search over the EM field, find_paths and extract_circuit against the sample graph.
- `demo/semantic_field_explainer.html`: full visual explainer for SemanticField — circuit diagram, layer stack, why EM, 1/2/multi-hop examples, test results. Open in any browser.
- `demo/eval_arch1_vs_arch2.py` and `demo/eval_qa_accuracy.py`: evaluation harnesses comparing plain traversal vs. ontology-guided traversal.

Install the demo extras and open any notebook:

```bash
uv sync --extra demo
marimo edit demo/demo_kg_connectors.py
```

Every demo runs offline against a small committed sample graph: no download,
no network, no external services.

## Data and acknowledgments

The ontology demo and the two evaluation harnesses run against a small
hand-authored sample of public movie facts (`demo/data/sample_kb.txt`) written
in the triple format of the MetaQA dataset (Zhang, Yuyu et al., "Variational
Reasoning for Question Answering with Knowledge Graph", AAAI 2018,
https://github.com/yuyuz/MetaQA). The sample is committed in this repo and is
not derived from the MetaQA dataset files. The notebooks call
`demo.data.ensure_data()` at startup, which builds the sample subgraph offline.
To run against the full MetaQA benchmark (licensed under
[CC BY 3.0](https://creativecommons.org/licenses/by/3.0/legalcode), not
redistributed here), see [`demo/data/README.md`](demo/data/README.md).

## Development setup

**Install uv** (if not already installed):
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

**Create virtual environment and install dependencies:**
```bash
uv venv
source .venv/bin/activate
uv sync --all-extras
```

**Run all checks with tox:**
```bash
uv tool install tox --with tox-uv
tox
```

### Run individual checks

```bash
pytest
ruff format --check --line-length 120 .
ruff check .
mypy --strict --ignore-missing-imports .
bandit -c pyproject.toml -r -q .
```

## Related repositories and documentation

- **[mloda](https://github.com/mloda-ai/mloda)**: The core library for open data access. Declaratively define what data you need, not how to get it. See [mloda.ai](https://mloda.ai) for an overview and business context and the [documentation](https://mloda-ai.github.io/mloda/) for detailed guides.
- **[mloda-registry](https://github.com/mloda-ai/mloda-registry)**: The central hub for discovering and sharing mloda plugins.
- **[Plugin development guides](https://github.com/mloda-ai/mloda-registry/tree/main/docs/guides/)**: How to build FeatureGroups, ComputeFrameworks, and Extenders.
- **[Claude Code skills](https://github.com/mloda-ai/mloda-registry/tree/main/.claude/skills/)**: Assisted plugin development for Claude Code users.
