Metadata-Version: 2.4
Name: snf-peirce
Version: 0.2.5
Summary: SNF/Peirce semantic query engine for retrieving data by WHO/WHAT/WHEN/WHERE/WHY/HOW instead of SQL joins
Author: Alexander Kern
License: Apache-2.0
Project-URL: Homepage, https://github.com/peirce-lang/snf-peirce
Project-URL: Repository, https://github.com/peirce-lang/snf-peirce
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: pandas>=1.5
Requires-Dist: duckdb>=0.9
Provides-Extra: roaring
Requires-Dist: pyroaring>=0.4; extra == "roaring"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: jupyter; extra == "dev"
Dynamic: license-file

# snf-peirce

Query data by WHO/WHAT/WHEN/WHERE/WHY/HOW instead of writing SQL joins.

Map your data to meaning once. Query it in plain language forever. No SQL. No physical schema knowledge. No joins.

The lens is the semantic schema — `suggest()` builds it from your data, you confirm it, and from that point on your data answers questions.

```python
from snf_peirce import suggest, compile_data, query
import pandas as pd

df       = pd.read_csv("my_collection.csv")
draft    = suggest(df)
draft.map("Artist", "who", "artist").nucleus("release_id", prefix="discogs:release")
lens     = draft.to_lens(lens_id="discogs_v1", authority="me")
compiled = compile_data(df, lens)

query(compiled, 'WHO.artist = "Miles Davis" AND WHEN.released = "1959"')
```

## What is SNF?

SNF (Semantic Normalized Form) is a data model and query protocol built around six universal dimensions:

| Dimension | Meaning | Examples |
|-----------|---------|---------|
| WHO | People, organisations, roles | author, publisher, attorney, artist |
| WHAT | Things, topics, identifiers | title, subject, ISBN, genre |
| WHEN | Dates, years, time periods | publication_date, year, date_added |
| WHERE | Places, locations, regions | publication_place, office, territory |
| WHY | Reasons, types, purposes | matter_type, audience, format_legal |
| HOW | Methods, formats, measurements | carrier_type, cmc, media_type |

Every fact in your data maps to one of these dimensions. Once mapped, you query by meaning — not by column name, table structure, or JOIN logic.

Peirce is the query language for SNF. Named after Charles Sanders Peirce, whose triadic sign relation maps directly onto the SNF record structure: Dimension → SemanticKey → Value.

## Install

```
pip install snf-peirce
```

Or clone and use directly:

```
git clone https://github.com/peirce-lang/snf-peirce
cd snf-peirce
pip install -e .
```

## Quick start

### From a CSV

```python
import pandas as pd
from snf_peirce import suggest, compile_data, query

df    = pd.read_csv("matters.csv")
draft = suggest(df)
print(draft)   # renders as a table in Jupyter

draft.map("attorney_name", "who",   "attorney")
draft.map("matter_type",   "why",   "matter_type")
draft.map("fiscal_year",   "when",  "year")
draft.map("office",        "where", "office")
draft.nucleus_composite(["client_id", "matter_id"],
                         separator="-", prefix="legal:matter")

lens     = draft.to_lens(lens_id="legal_v1", authority="firm")
compiled = compile_data(df, lens)

query(compiled, 'WHO.attorney = "Smith" AND WHERE.office = "Seattle"')
query(compiled, 'WHEN.year BETWEEN "2022" AND "2024"')
query(compiled, 'WHO.attorney = "Smith" OR WHO.attorney = "Jones"')
```

### From an existing lens-tool lens

Lenses created by the JavaScript lens-tool are directly compatible:

```python
from snf_peirce import load, compile_data, query
import pandas as pd

lens     = load("discogs_community_v1.json")
df       = pd.read_csv("discogs_sample.csv")
compiled = compile_data(df, lens)

query(compiled, 'WHO.author = "Miles Davis"')
query(compiled, 'WHEN.publication_date BETWEEN "1955" AND "1965"')
```

## Interactive shell

```
python shell.py csv://my_spoke_dir
```

```
peirce> WHO.author = "Miles Davis"
peirce> WHEN.publication_date BETWEEN "1955" AND "1965"
peirce> WHO.author = "Miles Davis" AND WHEN.publication_date = "1959"
peirce> \schema             — show all dimensions and fields
peirce> \schema WHO         — show fields in WHO with counts
peirce> \explain            — show execution plan for last query
peirce> WHO.<TAB>           — TAB-completes field names from substrate
peirce> \pivot              — toggle wide table view
peirce> exit
```

Shell features beyond the JS version:

- TAB completion — field names from the actual substrate
- `\explain` — execution plan with cardinality bars
- `\schema` — dimensions, fields, entity and value counts
- Discovery expressions — `WHO|*`, `WHAT|genre|*` work inline

## Guided setup (no coding required)

```
python guided_ingest.py
python guided_ingest.py mydata.csv
```

Walks through CSV → lens authoring → compilation → shell with prompts. No code required.

## Fetch from public APIs

### Scryfall (Magic: The Gathering)

```
pip install requests
python fetch_scryfall.py              # Guilds of Ravnica (default)
python fetch_scryfall.py war          # War of the Spark
python fetch_scryfall.py --list       # see available sets
```

```
peirce> WHAT.guild = "Dimir"
peirce> WHAT.color ONLY ("Blue", "Black")
peirce> WHAT.card_type = "Creature" AND HOW.cmc BETWEEN "1" AND "3"
peirce> WHAT.keyword = "Surveil"
peirce> WHO.artist = "Seb McKinnon"
peirce> WHAT|guild|*
peirce> WHO|artist|*
```

Note: `WHAT.guild = "Dimir"` and `WHAT.color ONLY ("Blue", "Black")` return different
result sets by design. Guild affiliation includes colorless cards (lands, artifacts)
that belong to Dimir thematically but have no color identity. The ONLY query returns
cards that are exclusively blue-black. Both answers are correct — they answer
different questions.

### Library of Congress catalog

```
python fetch_loc.py                          # default: jazz music
python fetch_loc.py "toni morrison"          # keyword search
python fetch_loc.py --subject "cooking"      # subject search
python fetch_loc.py --author "hemingway"     # author search
python fetch_loc.py --marc-file catalog.mrc  # from a .mrc file
```

```
peirce> WHO.author CONTAINS "Morrison"
peirce> WHAT.subject_topic CONTAINS "Jazz"
peirce> WHEN.publication_date BETWEEN "1950" AND "1970"
peirce> WHERE.publication_place = "New York"
peirce> WHAT|subject_topic|*
```

### Build your own fetcher

```python
from base_fetcher import SNFFetcher, fact, facts, facts_from_list

class MyAPIFetcher(SNFFetcher):
    lens_id   = "myapi_v1"
    set_name  = "My Dataset"
    spoke_dir = "myapi_spoke"

    def fetch(self):
        import requests
        return requests.get("https://api.example.com/data").json()["items"]

    def entity_id(self, item):
        return f"myapi:{item['id']}"

    def translate(self, item):
        eid = self.entity_id(item)
        return [
            *facts(
                (eid, "what", "title",  item.get("title")),
                (eid, "who",  "author", item.get("author")),
                (eid, "when", "year",   item.get("year")),
            ),
            *facts_from_list(eid, "what", "genre", item.get("genres", [])),
        ]

if __name__ == "__main__":
    MyAPIFetcher().run()
```

## Peirce query syntax

```
WHO.artist = "Miles Davis"                         equality
WHO.artist != "Miles Davis"                        not equal
WHEN.released > "1960"                             comparison
WHEN.released BETWEEN "1955" AND "1965"            range (inclusive)
WHAT.title CONTAINS "Blue"                         substring
WHO.artist PREFIX "Miles"                          starts with
WHAT.color ONLY "Red"                              exclusivity (single value)
WHAT.color ONLY ("Blue", "Black")                  exclusivity (exact set — no others)
NOT WHERE.office = "Seattle"                       negation
WHO.artist = "Miles Davis" AND WHEN.released = "1959"    AND (intersection)
WHO.artist = "Miles Davis" OR WHO.artist = "Coltrane"    OR (union)
(WHO.artist = "Miles Davis" AND WHEN.released = "1959")
OR
(WHO.artist = "John Coltrane" AND WHEN.released = "1964")  DNF

# Discovery (shell only — shows schema)
*                 all dimensions
WHO|*             all fields in WHO
WHO|artist|*      all values for WHO.artist
```

### ONLY — scalar vs set form

`ONLY "Red"` returns entities where the field has exactly one value and it is Red.

`ONLY ("Blue", "Black")` returns entities where the field contains Blue AND Black
and nothing else. This is the correct way to query for multi-valued exclusivity —
for example, Magic cards that are exactly blue-black with no other colors.

## MARC support

snf-peirce ships with the MARC Bibliographic Lens v1.0 — a complete field mapping
from MARC21 tags to SNF dimensions. Python port of MARCTranslator_v3.js. No extra
dependencies required.

```python
from parse_marc import parse_mrc
from marc_translator import MARCTranslator

records    = parse_mrc("catalog.mrc")
translator = MARCTranslator(source_id="loc")

for record in records:
    facts = translator.translate_record(record)
```

Key field mappings:

| MARC tag | → | Dimension | Semantic key |
|----------|---|-----------|-------------|
| 100$a | → | WHO | author |
| 245$a+$b | → | WHAT | title |
| 260$b / 264$b | → | WHO | publisher |
| 260$a / 264$a | → | WHERE | publication_place |
| 260$c / 264$c | → | WHEN | publication_date |
| 650$a | → | WHAT | subject_topic |
| 651$a | → | WHERE | subject_place |
| 600$a | → | WHO | subject_person |
| 655$a | → | WHAT | genre |
| 020$a | → | WHAT | isbn (nucleus) |

## Jupyter workflow

Results render as tables inline. Output is pandas.

```python
import pandas as pd
from snf_peirce import suggest, compile_data, query

df       = pd.read_csv("my_collection.csv")
draft    = suggest(df)             # renders as mapping table
compiled = compile_data(df, lens)  # renders substrate summary

query(compiled, 'WHO.artist = "Miles Davis"')  # renders result table inline

result   = query(compiled, 'WHEN.released BETWEEN "1955" AND "1965"', limit=None)
df_result = result.to_dataframe()   # pandas DataFrame — use anything
df_result.groupby("semantic_key")["value"].value_counts()
```

## File inventory

| File | Purpose |
|------|---------|
| parser.py | Peirce query parser — conformant with JS reference |
| lens.py | Lens authoring: suggest(), LensDraft, load(), save() |
| compile.py | Data compilation: compile_data(), Substrate |
| peirce.py | Query execution: query(), execute(), ResultSet |
| shell.py | Interactive query shell |
| guided_ingest.py | Guided setup script — no coding required |
| base_fetcher.py | Base class for API fetchers |
| fetch_scryfall.py | Scryfall / Magic: The Gathering fetcher |
| fetch_loc.py | Library of Congress catalog fetcher |
| marc_translator.py | MARC Bibliographic Lens v1.0 |
| parse_marc.py | Pure Python binary MARC / MARCXML parser |

## Running the tests

```
python -m pytest test_parser.py -v    # 59 tests — parser conformance
python test_lens.py                    # 69 tests — lens authoring
python test_compile.py                 # 57 tests — compilation and queries
python test_peirce.py                  # 57 tests — end-to-end query
python test_conformance.py             # 37 tests — cross-language proof
```

## Architecture

```
SNF / Peirce specification   open protocol (MIT)
           ↓
     snf-peirce              Python runtime / engine  ← this package
           ↓
       Reckoner               visual application for non-technical users
```

snf-peirce is Reckoner's data engine. It is also usable standalone for data
practitioners who want SNF in Python and Jupyter workflows.

## Note on Portolan

The `\explain` shell command displays a simplified execution plan showing
constraints ordered by estimated cardinality. This implements the same ordering
heuristic as Portolan's I1 algorithm for display purposes.

Full Portolan — schema validation, type checking, query rejection, composite
constraint reasoning — is a separate licensed component not included in this package.

## License

MIT. See LICENSE file.

SNF specification, Peirce query language, and MARC Bibliographic Lens v1.0 are
original works. Attribution required in source code and documentation. See project
licensing documentation for details on Portolan and Reckoner licensing.
