Metadata-Version: 2.4
Name: subsetsio
Version: 0.1.0rc1
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Dist: deltalake>=0.19
Requires-Dist: duckdb>=1.0
Requires-Dist: pyarrow>=16
Requires-Dist: pandas>=2.0
Requires-Dist: requests>=2.31
Requires-Dist: tqdm>=4.66
Requires-Dist: sqlglot>=25
Requires-Dist: fastapi>=0.115
Requires-Dist: pydantic>=2
Requires-Dist: uvicorn>=0.30
Requires-Dist: zstandard>=0.23
License-File: LICENSE
Summary: The subsets.io public data library: clone the catalog as Delta tables, search it with the production retrieval engine, and query it with local SQL — or serve it all over HTTP.
Keywords: data,datasets,statistics,delta-lake,open-data
Author: Nathan Snellaert
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://subsets.io
Project-URL: Repository, https://github.com/subsetsio/subsetsio

# subsetsio

The subsets.io public data library, local-first. One pip package that clones
the catalog of published Delta tables to your machine, searches it with the
production retrieval engine, queries it with guarded local SQL, and serves
it all over HTTP — no accounts, no API keys, no hosted service.

```bash
pip install subsetsio            # library + CLI into the active environment
subsets sync                     # clone the small tier (~3.5k tables, ~200 MB)
subsets serve                    # http://127.0.0.1:8080 over your mirror
```

Python 3.10 or newer; wheels are published for Linux, macOS, and Windows, so
there is nothing to compile.

Want the `subsets` command globally, isolated from your projects? Use
`uv tool install subsetsio` or `pipx install subsetsio` instead — same
package, same command; you'll still `pip install subsetsio` in any
environment where you want `import subsetsio`. If `subsets` isn't on your
PATH after a plain `pip install`, `python -m subsetsio` is the same CLI.

Data comes anonymously from the public bundles host over plain HTTPS
(`SUBSETS_BUNDLES_URL`, default `https://bundles.subsets.io`): the catalog
manifest, the metadata document, and tiered tar bundles rebuilt hourly from
whatever changed. Everything after `subsets sync` works offline.

## The CLI

```
subsets sync                   # clone / refresh the mirror (small tier)
subsets add --bundle medium    # widen the selection (--bundle large: the widest tier)
subsets add                    # print what this mirror is set to sync
subsets sync --dry-run         # see what would change
subsets sync --prune           # drop local tables outside the selection
subsets list                   # browse the catalog
subsets search "malaysia inflation"      # ranked search, in-process engine
subsets info dosm-cpi-headline-inflation # schema, license, attribution
subsets query 'SELECT * FROM "dosm-cpi-headline-inflation" LIMIT 10'
subsets serve [--host 127.0.0.1] [--port 8080]
subsets mcp                    # MCP server over stdio (Claude Desktop etc.)
```

Sync is bundle-based: sources are packed into three cumulative size tiers
(one tar per tier) at the natural breaks of their size distribution, so a
cold clone is a handful of GETs instead of tens of thousands. Re-syncing is
incremental — a bundle is only re-downloaded when its content changed, and
tables above the per-table size cap are not distributed.

`large` is the widest tier, not the whole database. We do not yet serve the
entire index — the published bundles are a slice of it, filled in
usefulness-adjusted size order (the most useful data per byte first), and
that slice widens over time. So `--bundle large` means "everything we
currently bundle", and re-syncing it later gets you more than it did before.

The mirror remembers its tier. `subsets add --bundle {small,medium,large}`
sets it (persisted at the mirror root, per mirror) and `subsets add` alone
prints it; every later `subsets sync` keeps that tier, so a bare sync never
silently narrows back to small. `sync --bundle X` still works standalone
and is remembered the same way — except under `--dry-run`, which previews
without choosing. Narrowing the selection deletes nothing on its own: the
tables from the wider tier stay until you ask for `sync --prune`.

`--prune` removes local tables that are no longer in the catalog selection.
It refuses to run when the catalog selection is empty, or when it would
take more than half the synced tables with it — both look like a degraded
catalog rather than housekeeping; `--force` says you meant it (a narrower
tier, a different `SUBSETS_BUNDLES_URL`).

`search`/`info`/`query` answer entirely from your machine and take `--json`
to emit the raw payload (the same shapes the local server speaks).
`subsets search` prints ranked dataset ids one per line, best hit first
(default 60) — ids are semantic slugs, so the bare list is the cheapest
candidate set an agent can consume. `--matched` annotates each id with its
match evidence: `values:myr@currency_code (6.7k)` means the literal `myr`
sits in that column's cells in that many rows (the `WHERE` clause writes
itself, and the CLI prints it as a `next:` suggestion), `title:`/`desc:`/
`text:` mark metadata hits, `~x` marks a token with no lexical trace — a
semantic match. Ids that explain themselves stay bare. Cell evidence covers
synced tables only (and only columns passing the value gates), so its
absence never means the value is absent. `subsets info <id...>` has the
detail (schema, license, attribution — several ids at once, and
`--query` to include the same match evidence). A typo-corrected query is
reported via a `corrected:` line (stderr in plain mode). No matches → empty
stdout, exit 1.
SQL is DuckDB dialect; dataset ids are the table names (double-quote them —
hyphens otherwise parse as subtraction), and the referenced datasets must
be synced first.

## Search — the production engine, in-process

`subsets search` (and `GET /search`) runs a compiled multi-lane retrieval
engine (Rust, shipped inside the wheel — see `rust/`) fused with weighted
reciprocal-rank fusion over a typo-corrected query: lexical lanes (curated
keywords and synthetic queries from the published metadata, IDF-weighted
text over titles and columns), two semantic lanes (static embeddings over
dataset descriptions and synthetic queries, so conceptual queries match
without exact words), and an exact-match lane over code-shaped values
(tickers, ISO codes, category labels) extracted from your synced tables —
so `subsets search AAPL` finds the table whose *cells* mention AAPL even
though its metadata never does. The whole ~26k-dataset catalog is
searchable regardless of how much is synced. The index persists under the
mirror (a value index plus the engine's state cache), so warm rebuilds
touch only what changed. The embedding model (~30 MB,
`minishlab/potion-base-8M`) downloads from HuggingFace on the first index
build and is cached locally — everything after that is offline.

## The local server

`subsets serve` binds `127.0.0.1:8080` by default and exposes five
endpoints over the mirror — no auth, errors as clean JSON
(`{"detail": "..."}`):

| Endpoint | What it does |
|---|---|
| `GET /search?q=&limit=&source=` | ranked datasets from the local index |
| `GET /datasets/{id}` | metadata + schema + attribution (license, source, source_url) |
| `POST /query` `{"sql": "..."}` | read-only SQL → `{columns, rows, row_count}` |
| `GET /catalog` | the synced catalog manifest, verbatim |
| `GET /health` | liveness: `{"status": "ok"}` |

SQL runs in an in-process DuckDB via `delta_scan` over your synced tables,
behind three guards: read-only statements only (writes, DDL, table
functions, file/URL access are refused with a 400), a 100,000-row result cap
(413 — aggregate or LIMIT for more), and a 60s execution timeout (504).

## The MCP server

`subsets mcp` speaks the Model Context Protocol over stdio — the same
engine as `serve`, exposed as three tools (`search`, `get_dataset`,
`execute_sql`) so MCP clients can use your mirror directly. No auth, no
network, stdlib-only transport. For Claude Desktop, add to
`claude_desktop_config.json`:

```json
{"mcpServers": {"subsets": {"command": "subsets", "args": ["mcp"]}}}
```

Tool failures (bad SQL, unknown dataset, nothing synced yet) come back as
`isError` tool results carrying the same messages the CLI prints, so the
model can read them and recover.

## The Python library

```python
import subsetsio

subsetsio.list_tables()                  # the catalog, as a DataFrame
subsetsio.search("refugee flows")        # ranked search, local engine
subsetsio.info("unhcr-population")       # schema, license, attribution (a dict)
df = subsetsio.read("unhcr-population")  # one table, as a DataFrame
df = subsetsio.query('SELECT * FROM "unhcr-population" LIMIT 10')  # local SQL
```

`read` serves the table from the local mirror (a pinned Delta table) —
repeated reads are milliseconds, and anything synced keeps working offline.

## For agents — the Claude skill

This repo ships a Claude skill that teaches agents the whole flow above
(install → sync → search/info/query → cite the upstream source). It lives
at `plugins/subsets/skills/subsets/SKILL.md` and versions with the package,
enforced by `tests/test_skill_parity.py`.

- **Claude Code**: add this repo as a plugin marketplace
  (`.claude-plugin/marketplace.json` at the repo root).
- **claude.ai**: `scripts/build-skill-zip.sh` builds the uploadable
  `dist/subsets-skill.zip`.

## How it works

The sync contract is the catalog manifest: every published table with its
pinned Delta version, path, and size, fetched in one GET. Bundles are the
only fetch path — the client downloads the tiers it asked for, extracts
exactly the tables whose catalog pin the bundle carries, and reports
anything the bundles don't cover (tables over the size cap, or newer than
the last bundle build). Cloned tables are valid Delta tables at the pinned
version — the log plus only the parquet files that version references.

Beside the catalog, `subsets sync` stores the host's `metadata.json`: per
dataset, the title, description, column schema, attribution (upstream
source, source_url, license), and the curated retrieval signals (keywords,
synthetic queries) that feed the search index. Both documents are parsed
strictly against a versioned contract (`subsetsio/models.py`) before anything
is persisted: every cataloged dataset must have a metadata entry, required
fields must be present and well-typed, and a violation fails the sync loudly
naming the document, dataset, and field — a bad publish never lands on the
mirror. Only the enrichment fields (description, keywords, queries, tags)
plus columns, license, and source_url may be absent, and they are still
type-checked when present.

## Configuration

| Variable | Meaning |
|---|---|
| `SUBSETS_BUNDLES_URL` | the bundles host — the public one by default (`https://bundles.subsets.io`), or a self-hosted mirror |
| `SUBSETS_MIRROR_DIR` | the mirror root shared by `read()`, `subsets sync`, and `subsets serve` (default `~/.subsets/data`) |

Environment variables are the whole user-facing configuration surface. The
one thing written to disk beside the data is the mirror's own selection
(`<mirror>/.selection.json` — the tier `subsets add` chose); deleting it
just resets the selection to small.

## License

MIT — see [LICENSE](LICENSE). The datasets it distributes carry their own
upstream licenses; `subsets info` reports each one, and `info --json` gives
you the field to cite.

