Metadata-Version: 2.4
Name: mustrd
Version: 0.8.0rc2
Summary: A Spec By Example framework for RDF and SPARQL, Inspired by Cucumber.
License: MIT
License-File: LICENSE
Author: John Placek
Author-email: john.placek@semanticpartners.com
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python
Classifier: Framework :: Pytest
Classifier: Natural Language :: English
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Utilities
Requires-Dist: Jinja2 (>=3.1.6,<4.0.0)
Requires-Dist: beautifulsoup4 (>=4.11.1,<5.0.0)
Requires-Dist: colorama (==0.4.6)
Requires-Dist: colorlog (>=6.7.0,<7.0.0)
Requires-Dist: coverage (==7.4.3)
Requires-Dist: edn-format (>=0.7.5,<0.8.0)
Requires-Dist: flake8 (==7.0.0)
Requires-Dist: multimethods-py (>=0.5.3,<0.6.0)
Requires-Dist: numpy (>=1.26,<3) ; python_version < "3.13"
Requires-Dist: numpy (>=2.1,<3) ; python_version >= "3.13"
Requires-Dist: openpyxl (>=3.1.2,<4.0.0)
Requires-Dist: pandas (>=2.0,<3.0)
Requires-Dist: pyshacl (>=0.30.0,<0.31.0)
Requires-Dist: pytest (>=7.2.0,<10)
Requires-Dist: rdflib (>=7.1.3,<8.0.0)
Requires-Dist: requests (>=2.28.2,<3.0.0)
Requires-Dist: tabulate (>=0.9.0,<0.10.0)
Requires-Dist: toml (>=0.10.2,<0.11.0)
Requires-Dist: tomli (>=2.0.1,<3.0.0)
Project-URL: Repository, https://github.com/Semantic-partners/mustrd
Description-Content-Type: text/markdown

# MustRD

**"MustRD: Validate your SPARQL queries and transformations with precision and confidence, using BDD and Given-When-Then principles."**

[![Coverage Badge](https://github.com/Semantic-partners/mustrd/raw/python-coverage-comment-action-data/badge.svg?sanitize=true)](https://github.com/Semantic-partners/mustrd/tree/python-coverage-comment-action-data)

## Why?

SPARQL is a powerful query language for RDF data, but how can you ensure your queries and transformations are doing what you intend? Whether you're working on a pipeline or a standalone query, certainty is key.

While RDF and SPARQL offer great flexibility, we noticed a gap in tooling to validate their behavior. We missed the robust testing frameworks available in imperative programming languages that help ensure your code works as expected.

With MustRD, you can:

- Define data scenarios and verify that queries produce the expected results.
- Test edge cases to ensure your queries remain reliable.
- Isolate small SPARQL enrichment or transformation steps and confirm you're only inserting what you intend.

## What?

MustRD is a Spec-By-Example ontology with a reference Python implementation, inspired by tools like Cucumber. It uses the Given-When-Then approach to define and validate SPARQL queries and transformations.

MustRD is designed to be triplestore/SPARQL engine agnostic, leveraging open standards to ensure compatibility across different platforms.

### What it is NOT

MustRD is not an alternative to SHACL. While SHACL validates data structures, MustRD focuses on validating data transformations and query results.

## How?

You define your specs in Turtle (`.ttl`) or TriG (`.trig`) files using the Given-When-Then approach:

- **Given**: Define the starting dataset.
- **When**: Specify the action (e.g., a SPARQL query).
- **Then**: Outline the expected results.

Depending on the type of SPARQL query (CONSTRUCT, SELECT, INSERT/DELETE), MustRD runs the query and compares the results against the expectations defined in the spec.

Expectations can also be defined as:

- INSERT queries.
- SELECT queries.
- Higher-order expectation languages, similar to those used in various platforms.

## Example

### Configuration File

You'll have a configuration `.ttl` file, which acts as a suite of tests. It tells MustRD where to look for test specifications and any triplestore configurations you might have:

```ttl
:test_example a :MustrdTest;
              :hasSpecPath "test/specs/";
              :hasDataPath "test/data/";
              :hasPytestPath "example";
              :triplestoreSpecPath "test/triplestore_config/triplestores.ttl";
              :filterOnTripleStore triplestore:example_test .
```

### Test Specification

In the directory specified by `:hasSpecPath`, you'll have one or more `.mustrd.ttl` files. These can be organized in a directory structure. MustRD collects them and reports results to your test runner.

```ttl
:test_example :given [ a :FileDataset ;
                       :file "test/data/given.ttl" ] ;
              :when [ a :TextSparqlSource ;
                     :queryText "SELECT ?s ?p ?o WHERE { ?s ?p ?o }" ;
                     :queryType :SelectSparql ] ;
              :then [ a :OrderedTableDataset ;
                     :hasRow [ :variable "s" ; :boundValue "example:subject" ;
                               :variable "p" ; :boundValue "example:predicate" ;
                               :variable "o" ; :boundValue "example:object" ] ].
```

And you will have a `'test/data/given.ttl'` which contains the given ttl. 

```ttl
example:subject example:predicate example:object .
```

### Running Tests

Run the test using the MustRD Pytest plugin:

```bash
poetry run pytest --mustrd --config=test/mustrd_configuration.ttl --md=render/github_job_summary.md
```

This will validate your SPARQL queries against the defined dataset and expected results, ensuring your transformations behave as intended.

You can refer to SPARQL inline, in files, or in Anzo Graphmarts, Steps, or Layers. See `GETSTARTED.adoc` for more details.

#### Named graphs

A quad `given` (TriG, NQuads, TriX) keeps its named graphs, so a `when` can read
one with a `GRAPH` clause. A `when` with no `GRAPH` clause still sees everything.

```turtle
must:given [ a must:FileDataset ; must:file "data/two-graphs.trig" ] ;
must:when  [ a must:TextSparqlSource ;
             must:queryText "SELECT ?v WHERE { GRAPH ex:graph-a { ?s ex:value ?v } }" ;
             must:queryType must:SelectSparql ] ;
```

A **`then` is compared as one flat union by default.** You should not have to say
which graph a triple is in just to assert that it exists, so a quad `then` passes
as long as the triples are all there, wherever they sit.

When the graph a triple lands in *is* the thing under test — a pipeline writing
each layer to its own graph — opt in:

```turtle
must:then [ a must:FileDataset ;
            must:matchNamedGraphs true ;
            must:file "data/expected.trig" ] .
```

Now the layout is part of the assertion: the right triples in the wrong graphs is
a failure, and the failure names the graphs that differ rather than handing you a
merged diff to work out which layer moved.

**Which query forms can produce named graphs is engine-specific, and it catches
people out.** An UPDATE always can — `INSERT { GRAPH ?g { … } }` is standard. A
CONSTRUCT cannot, in standard SPARQL 1.1: the template is triple patterns only, so
a `GRAPH` there is a *syntax error*, not an empty result. On RDFLib you get
`Expected ConstructQuery, found 'GRAPH'`.

Several engines extend it anyway — [Jena/ARQ](https://jena.apache.org/documentation/query/construct-quad.html)
since 3.0.1, Stardog via a graph template, and Anzo, whose CONSTRUCT clause takes
"a graph and triple template". The Jena and Stardog forms are recorded in the
[W3C SPARQL CG inventory of extensions](https://github.com/w3c-cg/sparql-dev/wiki/Inventory-of-existing-extensions-to-SPARQL-1.1);
standardising it is [w3c/sparql-dev#31](https://github.com/w3c/sparql-dev/issues/31).

That extension is what makes a **dry run** possible: swap `INSERT` for `CONSTRUCT`,
keep the template and `WHERE` as they are, and you get back the quads the update
*would* have written without writing them — then `must:matchNamedGraphs true`
asserts they land in the right graphs. mustrd's Stardog backend asks for TriG
rather than Turtle so those graph names survive the response.

Anzo already ships that rewrite as a button: for an INSERT or DELETE, the Query
Builder's [Dry Run](https://2024.help.altair.com/5.4/graphstudio/userdoc/query-builder-query.htm)
"runs a version of the query where INSERT or DELETE is replaced with CONSTRUCT",
reporting "additions or removals **per graph**". What that gives you once,
interactively, a spec gives you on every run with the expected graphs written down.

See `GETSTARTED.adoc` for the per-engine table.

#### When a spec fails

A failing SELECT names the binding that differs and what it differs by, on the
first line — before the table. The commonest failure of all, right shape and
wrong value, otherwise reports two identical shapes and leaves you to find the
one cell that matters:

```
Expected 1 row(s) and 1 column(s), got 1 row(s) and 1 column(s) — differs in: month (datatype: expected xsd:string, actual xsd:gYearMonth)

|    | ('month_datatype', 'expected')          | ('month_datatype', 'actual')                |
|---:|:----------------------------------------|:--------------------------------------------|
|  0 | http://www.w3.org/2001/XMLSchema#string | http://www.w3.org/2001/XMLSchema#gYearMonth |
```

A wrong term rather than a wrong type reads the same way, using the prefixes
your `given` declares:

```
Expected 2 row(s) and 3 column(s), got 2 row(s) and 3 column(s) — differs in: o (expected ex:object, actual ex:obj)
```

The ones that are hardest to see and easiest to make are two IRIs naming the
same thing under a different scheme or host. Those are named rather than
printed:

```
— differs in: s (scheme: expected http, actual https)
— differs in: s (host: expected company.dev, actual company.com)
— differs in: s (origin: expected http://company.dev, actual https://company.com)
```

The same ontology served from a dev host and a prod one, or over `http` on one
side and `https` on the other, agrees everywhere the eye lands. Printing both
IRIs makes it a spot-the-difference puzzle; eliding them makes it worse.

Only claimed when the origin really is the whole difference. If anything after
the host differs too you get both IRIs, because naming the host would send you
after the wrong thing.

Some details worth knowing, because they are what make it readable on real data:

- **The pair is only named when the two shapes match.** A row- or column-count
  mismatch already says what is wrong, and listing every column would be noise.
- **A binding whose value differs is named once.** Its datatype nearly always
  differs too; `(datatype: …)` is reserved for the case you cannot see otherwise
  — same text, different type.
- **A long value is elided around the difference, not from the start** — cutting
  the tail off two near-identical IRIs would show you the half they agree on and
  hide the half they don't:

  ```
  — differs in: s (expected <…o/the/thing/alpha>, actual <…o/the/thing/beta>)
  ```

- **A wide result falls back to bare column names.** Past a readable length the
  line has stopped being a summary, and the table below is the right tool.

#### What a run can produce

Every flag below works identically on `pytest --mustrd` and on `mustrd report`:

| Flag | You get | Detail |
| --- | --- | --- |
| *(none)* | pass/fail to the terminal | above |
| `--md=report.md` | Markdown summary, for a CI job summary or a PR comment | [Competency questions & coverage](#competency-questions--ontology-coverage) |
| `--viewer=report.html` | one self-contained HTML report — no server, no CDN | [The HTML report](#the-html-report) |
| `--cq` | competency-question table: which questions your tests answer | [Competency questions & coverage](#competency-questions--ontology-coverage) |
| `--term-coverage` | how much of your ontology the tests actually exercise | [Ontology term coverage](#ontology-term-coverage) |
| `--term-coverage-rdf=cov.ttl` | the same coverage as RDF (DQV + PROV) | [Ontology term coverage](#ontology-term-coverage) |
| `--results-rdf=run.ttl` | per-test results as RDF | [The HTML report](#the-html-report) |

```bash
# the usual pair: something to read in the terminal, something to attach to CI
pytest --mustrd --config=config.ttl --cq --term-coverage --md=report.md --viewer=report.html
```

`--viewer` already implies the coverage and competency-question graphs; `--cq` and
`--term-coverage` only affect the terminal and `--md` output.

#### Running on Windows

Windows is a supported, CI-tested platform — the matrix covers `windows-latest` on
Python 3.11, 3.12 and 3.13. Both front ends work the same as on Linux:

```powershell
pytest --mustrd --config=test\test_config_local.ttl --md=report.md
mustrd report --config test\test_config_local.ttl --viewer report.html
```

Two things differ in practice:

- **Non-ASCII output is handled for you.** The reports use `↳`, `▸`, `✅` and `❌`,
  and a Windows console still defaults to cp1252, where printing those raises
  `UnicodeEncodeError`. The CLI reconfigures its own streams to UTF-8 on startup, so
  this is not something you need to work around. If you drive mustrd from your own
  script and see an encoding error, `set PYTHONIOENCODING=utf-8` fixes it.
- **Paths in a config are resolved relative to the config file**, not the working
  directory, so backslashes and drive letters are fine and you can run from anywhere.

#### Running on locked-down Windows

Enterprise builds commonly block the bare `.exe` shims that pip installs into
`Scripts\`. The package installs fine, but the `mustrd` command won't start. Run the
module through the venv's interpreter instead:

```
python -m venv .venv
.venv/Scripts/python -m pip install mustrd

.venv/Scripts/python -m mustrd report --config config.ttl --viewer report.html
.venv/Scripts/python -m pytest --mustrd --config=config.ttl --md=report.md
```

`python -m mustrd` is exactly equivalent to the `mustrd` command — same entry point,
same flags. It exists for this.

#### Integrating with Visual Studio Code (vscode)
We have a pytest plugin.
1. Choose a python interpreter (probably a venv)
2. `pip install mustrd ` in it.
3. add to your settings.json
```json
    "python.testing.pytestArgs": [
        "--mustrd", "--md=junit/github_job_summary.md", "--config=test/test_config_local.ttl"
    ],
```
4. VS Code should auto discover your tests and they'll show up in the flask icon 'tab'.
![alt text](image.png)

Each `.mustrd.ttl` is a node in the tree, in the folder it actually lives in, with
one test under it per spec and triple store (`<spec>@<store>`). Running a single
one runs exactly that one. `:hasPytestPath` no longer shapes the tree — the
directories do — but it still filters, via `--pytest-path`.

#### Also worth installing: Mentor

If you are writing RDF in VS Code, get [**Mentor**](https://github.com/faubulous/mentor-vscode)
(`faubulous.mentor`, [mentor-vscode.dev](https://mentor-vscode.dev/), GPL-3.0). It is
the missing IDE for knowledge graphs, and it makes authoring mustrd specs markedly
less painful: syntax highlighting and validation for Turtle, TriG, N-Triples,
N-Quads, RDF/XML and SPARQL, browsable RDFS/OWL/SHACL/SKOS definition trees with
structural reasoning, workspace-wide autocomplete with prefix.cc lookup, go-to-definition
and cross-file references, prefix and IRI renaming, and a built-in triple store you
can run SPARQL against.

A mustrd spec is just Turtle, and Mentor treats it as such — jump straight from a
`must:fileurl` to the query it points at, or from a term in a spec to its definition
in your ontology. Nothing to configure on our side. Credit where it's due: it is an
excellent piece of work and not ours.

## Competency questions & ontology coverage

A competency question (CQ) is a first-class `cq:CompetencyQuestion` node (its
vocabulary is `cq:` = `https://mustrd.org/competencyQuestion/`) — it owns the
question (`cq:question`, a sub-property of `rdfs:label`) and *optionally* links
to the test(s) that answer it with `cq:cqSpec`. CQ nodes live in any
`.mustrd.ttl` in the suite:

```ttl
@prefix cq: <https://mustrd.org/competencyQuestion/> .

:rotterdamCountryCQ a cq:CompetencyQuestion ;
    cq:question "In which country is Rotterdam?" ;
    cq:cqSpec :test_example .        # optional — omit for a CQ with no test yet
```

Because the link is optional, you can record a CQ *before* writing its test; the
report lists such CQs as gaps (Test column "—").

Two opt-in report flags build on this, and compose:

- **`--cq`** adds a **Competency Questions** table (one row per CQ node — its
  linked test(s) and status) and a per-CQ breakdown. Needs no ontology.
- **`--term-coverage`** adds **ontology term coverage over all mustrd tests**
  (see below). Needs an ontology.

Plain `--md` (neither flag) is unchanged: it still writes the standard
test-results summary.

```bash
pytest --mustrd --config=config.ttl --md=report.md            # test-results summary
pytest --mustrd --config=config.ttl --cq --md=report.md       # + competency questions
```

### Ontology term coverage

`--term-coverage` reports **how much of your ontology your tests actually
exercise** — an overall percentage and a per-term table (to stdout, and the
`--md` file if given). Add `--cq` too and it also shows how much is backed by a
*competency question* (a stricter number) via a `CQ Term Coverage` column.

Tell MustRD which ontology to measure against with `:hasOntologyPath` in your
config — a file or a directory (scanned recursively), repeatable:

```ttl
:myTest a :MustrdTest ;
    :hasSpecPath     "specs/" ;
    :hasDataPath     "data/" ;
    :hasOntologyPath "ontology/" ;   # file or directory; repeat for several
    :filterOnTripleStore triplestore:RdfLib .
```

```bash
pytest --mustrd --config=config.ttl --term-coverage             # coverage to stdout
pytest --mustrd --config=config.ttl --term-coverage --cq --md=report.md
```

(If `--term-coverage` is set without `:hasOntologyPath`, MustRD fails early and
tells you exactly what to add.)

Coverage is **data-based**: a declared term counts as **covered** when a
*passing* test **populates it in input data** (as an instance type or asserted
predicate) — whether or not a query also names it. A term named *only* in a
query but never instantiated is **query-only** and does *not* count (the test
passes without it); it's often a sign a TBox axiom belongs in the ontology, not
the fixture — the report flags those under **⚠️ TBox axioms in test data**.
Terms that are only structurally referenced (the `rdfs:domain`/`rdfs:range` of a
used property, a superclass of a used class, or a metadata property such as an
`owl:AnnotationProperty`/`owl:OntologyProperty`) are reported separately as
**structural** terms and excluded from the percentage. Every term is classified
as *fully exercised*, *data-only*, *query-only*, *structural*, or *unused* — so
untested terms surface immediately. When you pass `--md`, the parent directory is
created automatically if it doesn't exist.

**RDF output.** `--term-coverage-rdf=coverage.ttl` writes the result as RDF (W3C
DQV + PROV) for a knowledge graph: quality measurements `computedOn` the ontology
IRI and its `owl:versionIRI` (value a decimal ratio), a per-term breakdown, and
quality issues — all with stable IRIs, no blank nodes.

See [`docs/ontology-term-coverage.md`](docs/ontology-term-coverage.md) for the
full definition and [`docs/examples/geography-example/report/term-coverage-example.md`](docs/examples/geography-example/report/term-coverage-example.md)
for sample output.

## The HTML report

`--viewer=report.html` writes **one self-contained HTML file**: no build step, no
CDN, no server. It carries the run's RDF, a Turtle parser and
[VanJS](https://vanjs.org) (~5KB, MIT, vendored inline), and builds
everything in the browser — a pass/fail/skip test tree with timings, the term
coverage table (classes nested by `rdfs:subClassOf`, properties under their
`rdfs:domain`), the competency questions, and the quality issues. Attach it to a
CI run, email it, or open it from disk.

```bash
pytest --mustrd --config=config.ttl --viewer=report.html
```

`--viewer` implies the coverage and competency-question graphs, so you don't need
`--term-coverage`/`--cq` as well (they only affect the terminal and `--md`
output). Coverage appears whenever the config declares `:hasOntologyPath`.

**Sources are embedded.** By default each spec's Turtle and the SPARQL it ran are
inlined into the page, syntax-highlighted, so the report is readable without the
files it was generated from — a path in a graph only resolves from the directory
the run happened in, which is no use in an emailed file or a CI artifact. Any file
reference in the report opens the embedded copy in place. `--no-viewer-sources`
turns this off for a smaller page.

The page is also a viewer for *any* mustrd graph: drop a `.ttl` or `.jsonld` from
`--term-coverage-rdf` / `--results-rdf` onto it, or point it at one with
`?ttl=path/to/run.ttl`. Drop several to compare or merge runs.

Related flags: `--results-rdf` / `--results-jsonld` write the per-test results
graph (every test, `passed`/`failed`/`skipped`, with timing) on its own;
`--term-coverage-jsonld` writes the coverage graph as JSON-LD. `--viewer-title`
sets the page title, and `--viewer-src-base` prefixes the page's source-file
links when it is served from somewhere other than the working directory.

[Live example report](https://mustrd.org/examples/geography-example/report/).

## The `mustrd` CLI

The pytest plugin is a front end, not the engine. Spec execution
(`mustrd.runner`) and reporting (`mustrd.reporting`) are plain libraries, and the
`mustrd` command drives them **without pytest** — same config file, same specs,
same reports:

```bash
mustrd run    --config config.ttl                      # run the specs, review results
mustrd report --config config.ttl --viewer report.html  # run + emit the reports
```

`config.ttl` is your own MustrdTest configuration — the same file `pytest
--mustrd --config=` takes. To try it against the worked example in this repo:

```bash
mustrd report --config docs/examples/geography-example/mustrd-config.ttl \
              --viewer report.html
```

Paths inside a config are resolved relative to *the config file*, so it can be
run from anywhere; the viewer's source links are relative to the working
directory (see `--viewer-src-base`).

`mustrd report` takes the same reporting flags as the plugin (`--md`,
`--term-coverage`, `--cq`, `--term-coverage-rdf`, `--results-rdf`, `--viewer`, …)
plus `--ontology` to override `:hasOntologyPath`. Both exit non-zero if any spec
does not pass.

## When?

MustRD is a work in progress, built to meet the needs of our projects across multiple clients and vendor stacks. While we find it useful, it may not meet your needs out of the box.

We invite you to try it, raise issues, or contribute via pull requests. If you need custom features, contact us for consultancy rates, and we may prioritize your request.

Contributing: see [CONTRIBUTING.md](CONTRIBUTING.md) for setup, the two test suites, and the traps.

## Releasing

Maintainers: releases are cut by pushing a version tag (`git tag 0.7.5 && git push origin 0.7.5`). See [RELEASING.md](RELEASING.md) for the full flow, including beta/candidate releases.

## Support

Semantic Partners is a specialist consultancy in Semantic Technology. If you need more support, contact us at info@semanticpartners.com or mustrd@semanticpartners.com.



