Metadata-Version: 2.4
Name: fd-open-data-protocol
Version: 0.3.0
Summary: The open-data datasource protocol: a manifest contract a datasource exposes to be ingested by fd-open-data-mcp (or any consumer).
Author: FindDataOfficial
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"

# fd-open-data-protocol

**English** | [中文](README.zh-CN.md)

The **open-data datasource protocol**: a manifest contract a datasource exposes
(datasource + functions + columns + concept hints + fetch reference) so that
`fd-open-data-mcp` - or any consumer - can ingest it via `register_datasource`.

**Ship one manifest file -> the datasource is added. No consumer-side wiring.**

## One-click install

This library is a dependency of `fd-open-data-mcp` (pulled in transitively). To
install the **entire** finddata stack (hub + every datasource + ontology DB):

```bash
pip install "fd-open-data-mcp[data]" fd-polygon fd-cn-report

fd-open-data-mcp migrate \
  && fd-open-data-mcp import-catalog \
  && fd-open-data-mcp consume-concepts \
  && fd-open-data-mcp propose-bindings \
  && fd-open-data-mcp seed-entities \
  && fd-open-data-mcp generate-schedules \
  && fd-open-data-mcp register-discovered

fd-open-data-mcp serve
```

## The manifest

A YAML/JSON file (or a Python module exposing `CATALOG`):

```yaml
version: "1"
name: my-source
label: My Source
ranking_seed: [0.7, 0.7]            # [quality, accessibility] heuristic seed
functions:
  - command: get_data
    frequency: daily
    parameters: [{name: symbol, type: str, required: true}]
    columns:
      - {name: close, type: float, frequency: daily}
concepts:                           # column -> concept hints (measure/entity_type here)
  - {column: close, concept: price.close, entity_type: stock, unit: currency, frequency: daily}
fetch:
  runner: my-source                  # built-in runner name, OR module: "pkg.mod:run"
```

See `examples/example_stock.yaml` (declarative) and `examples/example_macro.py`
(a `DataProvider` class with `run()`).

## Load + validate

```python
from fd_open_data_protocol.loader import load_catalog
manifest = load_catalog("examples/example_stock.yaml")
print(manifest.name, len(manifest.functions))
```

`load_catalog` accepts a YAML/JSON file path, a `.py` file exposing `CATALOG`,
a `"pkg.mod"` module path, or a dict.

## Register with fd-open-data-mcp

```bash
fd-open-data-mcp register-datasource examples/example_stock.yaml
```

or the MCP tool `register_datasource(path)`.

## Publish a datasource from another project

In your datasource package's `pyproject.toml`:

```toml
[project.entry-points."fd_open_data_mcp.datasources"]
my-source = "my_pkg.catalog:CATALOG"
```

`pip install my-pkg` -> fd-open-data-mcp auto-registers it on `import_catalog`.

## Schema

- **`DatasourceManifest`**: name, label, source_url, scanner_mode, ranking_seed, functions[], concepts[], entities[], entity_definitions[], relationships[], fetch.
- **`FunctionSpec`**: command, category, description, parameters[], columns[], frequency, verified.
- **`ColumnSpec`**: name, type, description, meaning, semantic_type, `frequency` + `datasource` (column-level).
- **`ConceptHint`**: column, concept, `entity_type`, `measure`, unit, frequency, confidence, `concept_family` (optional).
- **`EntitySpec`**: entity_type, coverage ("universe"|"explicit"), codes[] (for explicit coverage).
- **`Entity`**: entity_type, code, name_en, name_zh, metadata{}, relationships[].
- **`EntityRelationship`**: target_entity_type, target_code, relation_type, confidence, metadata{}.
- **`RelationshipSpec`**: relation_type, source_entity_type, target_entity_type, resolver_module.
- **`FetchRef`**: runner (built-in name) | module (`"pkg.mod:func"`).

`measure` + `entity_type` are **concept-level** (disambiguate GDP-nominal vs
GDP-PPP; stock close vs fund NAV). Column-level `frequency`/`datasource` support
composite functions whose columns come from different sources at different cadences.

## Entity Definitions

The protocol supports two ways to declare entities:

### 1. Coverage Declaration (`entities[]`)

Declares which entity types the datasource covers:

```yaml
entities:
  - entity_type: stock
    coverage: explicit
    codes: [AAPL, MSFT, GOOGL]
```

- `coverage: "universe"` - datasource can fetch data for all entities of this type
- `coverage: "explicit"` - datasource only covers the listed codes

### 2. Entity Metadata (`entity_definitions[]`)

Defines canonical entity metadata (names, attributes, relationships):

```yaml
entity_definitions:
  - entity_type: stock
    code: AAPL
    name_en: Apple Inc.
    name_zh: 苹果公司
    metadata:
      exchange: NASDAQ
      sector: Technology
    relationships:
      - target_entity_type: industry
        target_code: gics_10
        relation_type: belongs_to
```

When included, entities are registered in the ontology database during `register_datasource()`.

#### Reserved metadata key: `external_ids`

An entity definition may anchor itself in an external entity graph under the
reserved metadata key `external_ids`. Registration persists each entry as a
per-source identifier, so consumers can resolve the external id back to the
local entity.

```yaml
entity_definitions:
  - entity_type: country
    code: CN
    name_en: China
    metadata:
      external_ids:
        wikidata: Q148          # Wikidata QID (Q followed by digits)
        datacommons: country/CHN  # Data Commons DCID
```

Supported anchor sources are `wikidata` (QID, validated as `Q<digits>`) and
`datacommons` (non-empty DCID). A malformed or unknown anchor is logged and
skipped — it never fails registration.

### Canonical Entity Types

The entity-type vocabulary is defined by
[`fd_open_data_protocol/vocabulary/entity_types.yaml`](fd_open_data_protocol/vocabulary/entity_types.yaml)
(14 types) and is the authority for manifest validation:

| Type | Description | Example IDs |
|------|-------------|-------------|
| `country` | ISO codes | CN, US, JP |
| `city` | Municipalities | beijing, shanghai |
| `stock` | A-shares | 600000.SH, 000001.SZ |
| `fund` | ETFs/funds | etf_code, fund_code |
| `bond` | Bonds | bond_code |
| `index` | Indices | SH000001, SZ399001 |
| `future` | Futures | cu2412, rb2401 |
| `crypto` | Cryptocurrencies | btc, eth |
| `organization` | General orgs | org_code |
| `industry` | Classifications | shenwan_1_01, gics_10 |
| `exchange` | Exchanges / venues | XNYS, XNAS, SSE |
| `company` | Public companies | AAPL, TSLA |
| `commodity` | Raw goods | steel, oil, grain |
| `person` | Humans (fund managers) | person_code |

## Semantic vocabulary

The protocol ships the publishable semantic core of FindData as versioned YAML
under [`fd_open_data_protocol/vocabulary/`](fd_open_data_protocol/vocabulary/):

| File | Registry |
|------|----------|
| `entity_types.yaml` | entity types (the manifest vocabulary above) |
| `concepts.yaml` | concept families + their seeded Variables |
| `qualifiers.yaml` | qualifiers (nominal_current, real_constant, ppp, per_capita, growth, ...) |
| `units.yaml` | units of measure (UN/CEFACT codes + ISO 4217 currencies) |
| `relation_types.yaml` | entity relationship types |
| `event_types.yaml` | event types (empty in this release) |
| `crosswalks/*.yaml` | Variable ↔ external-vocabulary assertions |

The vocabulary files ship as package data (`0.3.0`+), so `load_vocabulary()`
works from an installed wheel with no checkout on disk.

```python
from fd_open_data_protocol.vocabulary import load_vocabulary, validate_vocabulary

vocab = load_vocabulary()                    # shipped files
vocab = load_vocabulary("/path/to/dir")      # or an explicit directory
vocab.version                                # release version of the file set
vocab.get("concept", "GDP").uri              # https://schema.finddata.tech/concept/GDP
vocab.entity_type_ids()                      # the manifest entity-type vocabulary
vocab.seeded_variables()                     # Variables a family declares
errors = validate_vocabulary()               # [] when the file set is valid
```

Every entry carries a stable URI of the form
`https://schema.finddata.tech/<kind>/<Id>`, where `<kind>` is one of
`entity-type`, `concept`, `qualifier`, `unit`, `relation-type`, `event-type`.

**Stability rules.** A URI is never reassigned to a different meaning and
entries are never deleted. Retirement is expressed with a `superseded_by`
pointer to the successor entry, which keeps the URI resolvable. The file set
carries one release `version`, and the loader rejects a file set whose files
disagree.

## Concept families and Variables

Concepts are two-level. A **concept family** (`fd:GDP`, `fd:Population`) is the
small curated vocabulary from `concepts.yaml`; a **Variable** is the qualified
combination `(code, entity_type, measure, unit, frequency)` — the identity a
datasource actually binds columns to.

A manifest concept hint may point at its family with the optional
`concept_family` field, which the loader validates against `concepts.yaml`:

```yaml
concepts:
  - {column: close, concept: price.close, entity_type: stock, unit: currency,
     frequency: daily, concept_family: PriceClose}
```

Omitting it preserves the previous behaviour — no family validation occurs, and
the consumer derives a family from the concept code.

## Manifest Declaration Requirement

**Every fd-* datasource package MUST declare a `DatasourceManifest` via one of the following mechanisms:**

1. **Python module**: Expose a `CATALOG` dict in a module (e.g., `catalog.py`) that conforms to the `DatasourceManifest` schema
2. **YAML/JSON file**: Place a manifest file at the package root (e.g., `catalog.yaml` or `catalog.json`)
3. **Entry-point declaration**: Register the manifest path in `pyproject.toml` under `[project.entry-points."fd_open_data_mcp.datasources"]`

The declaration **SHALL** be discoverable by `fd-open-data-mcp`'s auto-discovery mechanism (`register-discovered` command). Packages without a CATALOG declaration **SHALL NOT** be considered compliant with the fd-open-data-protocol.

### Recommended Package Structure

```
my-datasource/
├── pyproject.toml          # declares entry-point
└── my_pkg/
    ├── __init__.py
    └── catalog.py          # exposes CATALOG = { ... }
```

### Entry-Point Declaration

In your `pyproject.toml`:

```toml
[project.entry-points."fd_open_data_mcp.datasources"]
my-source = "my_pkg.catalog:CATALOG"
```

After `pip install my-pkg`, the package is automatically discoverable:

```bash
fd-open-data-mcp register-discovered
```

### Auto-Discovery Flow

1. **Install package** → `pip install my-datasource`
2. **Entry-point registered** → setuptools records `my-source = "my_pkg.catalog:CATALOG"`
3. **Auto-discover** → `fd-open-data-mcp register-discovered` scans all entry-points
4. **Load manifest** → `load_catalog()` validates and parses the CATALOG dict
5. **Register to ontology** → `register_datasource()` upserts sources/functions/columns/concepts

### Compliance Checklist

Before publishing a new datasource package, ensure:

- [ ] Package exposes a `CATALOG` dict or manifest file
- [ ] `pyproject.toml` declares entry-point under `fd_open_data_mcp.datasources` group
- [ ] CATALOG conforms to `DatasourceManifest` schema (version, name, label, functions[], concepts[], fetch)
- [ ] `load_catalog()` can successfully parse the manifest
- [ ] `fd-open-data-mcp register-discovered` discovers and registers the package

### Working Examples

- **fd-world**: `fd_world/catalog.py` + entry-point in `pyproject.toml`
- **fd-cn-gov**: `fd_cn_gov/catalog.py` + entry-point in `pyproject.toml`
- **fd-cn-report**: `catalog.py` + entry-point in `pyproject.toml`

## Template

Copy `template/datasource.template.yaml` (declarative) or
`template/provider_template.py` (a `BaseDataProvider` class with `run()`).
