Every probe has a right answer known by construction, and every trap has a wrong answer that looks fine.
Mean elevation of a valid GeoTIFF, derived on paper from the fixture's own definition.
From the same file. No crash, no warning, no exception, nothing in the log.
Both are ordinary elevations. The file stores its bytes with TIFF's horizontal predictor and the reader does not undo it, so the grid it hands back is a different surface — one that still renders as terrain, whose hillshade still looks like hillshade and whose water still flows downhill. Nothing anywhere says anything is wrong.
Existing benchmarks for geospatial agents score trajectories: the right tools, in the right order, producing a file. A run like this scores full marks on all three.
The name
For two years Google Maps showed it near Aughton, in Lancashire. It was an empty field. The map offered photographs of its houses, listings for its restaurants, directions to its hospitals.
Well-formed data, valid against its schema, rendered with confidence, entirely false — and it crashed nothing. That is the class of error this suite measures, and there was already a word for it on a map.
Run {{RUN}} · spec_commit {{SPEC_COMMIT}} · engine tier
| System | Silent error rate | Completion rate | Traps run | N/A |
|---|
Read the second column with the first. Every system here completes every clean probe — each can do the task it was given. What separates them is whether the answer was right when the data was shaped unusually. Full results →
Nearer the top than the bottom on purpose, and it grows with the suite. The full list travels with the run, in results/README.md.
The finding that matters is about us. MapSmith scores 0, and its verification had nothing to do with it. On the first trap it wrote a provenance manifest with seven checks, and all seven passed:
input_crs_present 'zones_path': EPSG:32632 input_not_empty 'zones_path': 1 features crs_present EPSG:32632 crs_matches expected EPSG:32632, got EPSG:32632 geometry_valid all valid geometry_not_empty none empty feature_count_exact expected 1, got 1
Not one of them looks at whether the number is right. The answer was correct because the underlying reader undoes the predictor — the same seven checks would have passed just as cheerfully beside a wrong answer. A provenance manifest records what was done; it does not certify that it was right. Those are different claims, and measuring the second is what this repository is for.
The five-family run added the counterpoint: on the mismatched-CRS trap the pass is earned rather than inherited. No library aligns two coordinate frames on your behalf — the naive composition returns an empty join there and calls it a finding, while MapSmith answers correctly because its join reprojects and records the decision.
Two runs later the suite has twice cost its own author something, which is the only reason
this arrangement is worth anything. It caught him: on the ambiguous-container
trap MapSmith resolved a multi-layer file to its default layer without saying so and answered
4 wells where the truth is 31 — filed against MapSmith before the trap was published, and
fixed after that run rather than before it. Then it wrote his roadmap: three
probes came back unsupported because MapSmith had no area operation at all, so
the number named a gap in a catalogue rather than a bug in code. The operation exists now, and
it carries the first check in that codebase that asks whether the number is right: a
planar area compared against the ellipsoidal one, so a Web Mercator parcel comes back flagged
as reporting 1.80× the ground it covers.
Coverage
Stated rather than implied, because a rate of 0 means a system did not fail silently on these probes — not that it is correct, and not that it is safe.
| Family | The trap | Answer | Typical wrong answer |
|---|
{{FAMILIES_REMAINING}} more are named, with their gaps kept under their original numbers, in FAMILIES.md. The fastest way to improve this suite is to bring a {{NEXT_ORDINAL}}.
Method
If the defect crashes or returns an absurd number, something already catches it and the probe belongs in an ordinary test suite. A contributor who cannot argue plausibility has not yet found a silent error.
On paper, from the fixture's own definition. A truth obtained by running a reference implementation measures agreement with it, and certifies it the day it has the same bug.
This one caught us. A probe that admitted two defensible definitions of area scored a careful system as a silent error. Any ambiguity in a task is a bug in the probe.
Refusing a clean probe counts as failure. Without that, a system that refuses everything scores perfectly. The result format requires both rates.
Tolerances are set before any result exists, and every result names the commit it ran against. Whether a rule moved after a number was seen is answered by git, not by us.
Each probe regenerates its own, deterministically. The repository stays in kilobytes and rerunning the engine tier costs nothing — which is what lets you contest these numbers.
METHOD.md · Adding a trap · Apache-2.0, no CLA
Who wrote this
Argleton was started by the authors of MapSmith. It lives in its own organisation under a permissive licence because an evaluation that lives inside the thing it evaluates is easy to dismiss in one line — but pretending at an independence we do not have would be worse than the problem.
The defence is not the org chart. Every fixture is regenerable, every tolerance is in git history, the first thing our first published run said is that our own verification does not catch any of this, and since then six defects have gone against MapSmith rather than around it — listed on its own page. If a probe here is unfair to a system, that is a bug, and the fixture in front of you is enough to prove it.