The Check Registry

How the panel runs

One registry, three ways of producing a finding. Everything downstream — the HTML report, the CSV, the Cucumber features, the severity gate that sets the exit code — consumes a single row type, so where a finding came from never leaks into how it is reported.

registry.json is the single source of truth. It holds no logic: each entry is an id, a category, a severity, the prose shown to whoever reads the finding, and the Cucumber feature/scenario names. A check's implementation lives elsewhere, and there are three kinds:

Discovery is by directory walk, not by manifest. sparql_runner globs sparql/**/*.rq and shacl_runner loads every shapes/*.ttl, so adding a file is enough to add a check and deleting one is enough to remove it. The registry entry supplies the prose; a query with no matching entry, or an entry with no implementation, is the kind of drift the coverage tests exist to catch.

The result contract

Both engines are made to speak SHACL's own vocabulary, which is what lets their output merge. A portable SPARQL check is a CONSTRUCT that builds a sh:ValidationResult by hand:

CONSTRUCT {
  _:r a sh:ValidationResult ;
    sh:resultSeverity sh:Warning ;
    sh:focusNode ?p ;
    sh:sourceConstraintComponent oq:STR-003 ;
    sh:resultMessage ?msg .
}
WHERE {
  { ?p a owl:ObjectProperty } UNION { ?p a owl:DatatypeProperty }
  FILTER NOT EXISTS { ?p rdfs:domain ?d }
  FILTER NOT EXISTS { ?p rdfs:range ?r }
  BIND(CONCAT("Property ", STR(?p), " declares neither domain nor range.") AS ?msg)
}

Four properties are mandatory: sh:resultSeverity, sh:focusNode, sh:resultMessage, and sh:sourceConstraintComponent oq:<id> — that last one is how a bare CONSTRUCT result gets tied back to its registry entry. Add sh:resultPath and sh:value when there is a natural predicate or offending value. Binding several of either on one result is legitimate and sometimes the honest thing to do: LOG-004 names both inverses it is complaining about. They are sorted and joined on the way in, so a finding renders identically however the engine happened to order them.

Everything then normalises to one ResultRow:

ResultRow(check_id, category, title, severity,
          focus_node, path, value,
          message, remediation, sources)

Rows are deduplicated on (check_id, focus_node, path, value). When both engines find the same thing, one row survives carrying sources: ["shacl", "sparql"] — which is precisely the signal described next.

Why two formulations of the same check

Eighteen checks are written twice: once as a SPARQL CONSTRUCT, once as a SHACL shape. The duplication is deliberate. Running both and merging on that dedup key means a check that fires from one engine but not the other shows up as a row with only one entry in sources — a visible sign the two formulations have drifted apart, rather than a silent disagreement about what the check means.

--engine selects what actually runs:

The speed gap is not subtle: measured over a real 3,300-triple ontology, the same ~50-check pass took about 193 seconds under pyshacl against about 27 seconds for the portable layer, because pyshacl spends most of its time on Python-level shape traversal layered on top of the same SPARQL execution. Cross-validate in CI; use sparql or native+sparql while iterating.

The catalogue

All 56 checks, grouped by category. Severity is the registry default and can be overridden per project; the badges say which formulations exist for each check and where to find them.

A note on the closure-safe split

Four logical checks live in sparql/logical/closure-safe/ rather than beside their siblings, because the reasoning layer re-runs only that subdirectory (plus sparql/reasoning/) against the owlrl deductive closure. Both directories are discovered normally by the ordinary checks stage; the split changes only what gets re-run after inference.

The distinction is real, not organisational. LOG-001, LOG-002, LOG-004 and LOG-005 describe genuine contradictions that stay contradictions whether the triples producing them were asserted or entailed — catching those post-closure is the whole point of the reasoning pass. The others describe how the ontology was authored, and become meaningless once inference has run: LOG-003 flags a redundantly authored equivalentClass/subClassOf pair, but OWL2 RL entails that reciprocal subClassOf from every equivalentClass unconditionally. Run post-closure against a real gist-importing ontology, it fired on 59 of 59 axioms, none of them authored redundantly. LOG-006 and LOG-007 fail the same way through rdfs:subPropertyOf inheriting a superproperty's domain and range.

Adding a check

Pick the lightest of the three kinds that fits. Most checks are a graph pattern, and a graph pattern is a SPARQL check.

A portable SPARQL check

1. Pick an id and category

   Ids follow <PREFIX>-<NNN>. In use today: STR structural, LOG logical, QUA quality, EFF efficiency, STY style, DAT data, REA reasoning, CNF conformance. A new category is fine — add it to the registry and to CATEGORY_TITLES in docs/generate_checks_md.py.

2. Add the registry entry

   {
     "id": "STY-006",
     "category": "style",
     "metric": "short description of what is measured",
     "default_severity": "Warning",
     "title": "One-line summary",
     "description": "What condition is being flagged, in prose.",
     "remediation": "What the reader should do about it.",
     "cucumber_feature": "Naming Style",
     "cucumber_scenario": "One sentence, phrased as an expectation that should hold"
   }

   The scenario line is phrased as the expectation, not the failure — it is rendered as a Gherkin Then, so it reads as something that should be true.

3. Write sparql/<category>/<id>.rq

   Follow the result contract above. Keep it self-contained: its own PREFIX declarations, no reliance on external state. Copy a neighbour in the same category as a starting point.

4. Run it against a graph that should fire, and one that should not

   from rdflib import Graph
   g = Graph(); g.parse("fixture.ttl", format="turtle")
   q = open("sparql/style/STY-006.rq", encoding="utf-8").read()
   print(len(list(g.query(q).graph)))

   A query that looks right but matches nothing against a fixture that should trigger it is a bug, not a pass. This suite's own REA-001 shipped with exactly that: a FILTER(STR(?c1) < STR(?c2)) assuming owl:disjointWith gets symmetrised by reasoning, which it does not.

5. Regenerate the docs

   python docs/generate_checks_md.py

   Nothing else to wire. The runners discover the file, the registry resolves the id, and the check appears in every table, plot and Cucumber feature automatically.

Adding a SHACL twin

Optional but recommended, since it is what earns the cross-validation signal. Put the shape in shapes/<category>.ttl. Prefer native SHACL core constraints — sh:minCount, sh:pattern, sh:or, sh:disjoint, property paths — where the check maps onto them cleanly; otherwise use sh:sparql with a sh:select mirroring the .rq file's WHERE clause:

oq:STR-003
  a sh:NodeShape ;
  rdfs:label "STR-003: property missing both domain and range" ;
  sh:severity sh:Warning ;                 # on the shape, never on the constraint
  sh:target [
    a sh:SPARQLTarget ;
    sh:select """SELECT ?this WHERE { ?this a owl:ObjectProperty }""" ;
  ] ;
  sh:sparql [
    a sh:SPARQLConstraint ;
    sh:message "{$this} declares neither domain nor range." ;
    sh:select """SELECT $this WHERE { ... }""" ;
  ] .

Naming the shape exactly oq:<id> is all the tagging it needs. If one check needs several node shapes, put oq:checkId "<id>" on each instead.

A native Python check

Reach for this only when there is genuinely no single graph to pattern-match — OWL2 profile membership, an external DL reasoner's verdict, comparing a graph against a separate ontology's declarations, or diffing two ontology versions. The conformance family is the clearest example: CNF-001 through CNF-005 cannot be SPARQL checks because they need two graphs, one supplying the declarations and one supplying the usage.

1. Add the registry entry anyway

   Same fields as any other check. This is what lets the finding flow through the same report layer as everything else. There is deliberately no separate native-check registry.

2. Return ResultRow objects directly

   Construct them with your check_id and category, plus whatever focus_node, path, value, message, remediation and sources apply. conformance_to_rows in dataquality/data_quality.py is the pattern to copy.

3. Wire it into a pipeline stage

   Append the rows to the relevant stage's StageResult.rows, then regenerate the docs.

Running your own panel

You do not need a checkout of the suite, or to edit anything inside site-packages, to change which checks run. checks, data and run each accept --registry, --shapes and --sparql, independently overridable and defaulting to the installed package's own resources.

# find the installed defaults and take a copy you can edit
python -c "from ontology_suite import config; print(config.PACKAGE_RESOURCES)"
cp -r <the path printed above> my-checks

# remove a check: delete its query -- discovery is a directory walk, not a manifest
rm my-checks/sparql/quality/QUA-004.rq

# add one: a registry entry plus a .rq file, exactly as above

# point the run at your copy
ontology-quality-suite checks --ontology domain.ttl   --registry my-checks/registry.json   --shapes   my-checks/shapes   --sparql   my-checks/sparql

The three flags are independent, so a project can keep the stock shapes and SPARQL while overriding only the registry — which is how you re-severitise a check for one project without touching its logic. Change default_severity in your copy and the finding keeps its id, prose and implementation while moving between Violation, Warning and Info. Since --fail-on gates the exit code on severity, that is also how you decide what breaks a build.

A minimal worked example ships in the repo: examples/acme_robotics/custom_checks/ holds a project-local registry with a single check, ACM-001, and its query — a complete illustration of the smallest thing that works.

Pitfalls worth knowing

Vocabulary you do not declare yourself

Checks that ask "is this term declared?" need to know which terms nobody declares locally. Built-in RDF, RDFS and OWL2 vocabulary is assumed axiomatically — no ontology asserts owl:Restriction a owl:Class. Omitting that exemption produced 149 false "undefined class" findings against one vehicle ontology, one per anonymous restriction. The same applies to annotation properties nobody re-declares: rdfs:label, rdfs:comment and the SKOS lexical properties. A missing skos:prefLabel exemption once flooded every SKOS-labelled taxonomy with false undeclared-property findings.

Conventions that are not rdfs:domain and rdfs:range

A structural check that hardcodes one modelling convention will fire on every ontology using a different one. STR-003 accepts gist-style domainIncludes and rangeIncludes as satisfying "has a domain and range", matched by local name rather than by a hardcoded namespace, since gist has published under more than one over the years.

Findings that only make sense before inference

Before adding a check to sparql/logical/, decide whether it survives the closure. If it describes how axioms were authored rather than what they entail, it belongs outside closure-safe/.

Nested property shapes lose their check id

pyshacl reports a property-constraint violation's sh:sourceShape as the nested blank-node property shape, not the enclosing node shape carrying the oq:checkId annotation. Id resolution walks up via sh:property to compensate. Before it did, 435 of 894 findings against a real ontology arrived with no check id, category or remediation — every native SHACL-core finding in the suite.