Metadata-Version: 2.4
Name: soul-schema
Version: 0.1.0
Summary: Auto-generate semantic layers for any database using LLMs. Metadata only.
Project-URL: Homepage, https://github.com/menonpg/soul-schema
Project-URL: Repository, https://github.com/menonpg/soul-schema
Author-email: Prahlad Menon <menon.prahlad@gmail.com>
License: MIT
Keywords: data-catalog,database,llm,metadata,semantic-layer,text-to-sql
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: click>=8.0
Requires-Dist: requests>=2.28
Requires-Dist: rich>=13.0
Requires-Dist: sqlalchemy>=2.0
Provides-Extra: all
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: openai>=1.50; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: openai
Requires-Dist: openai>=1.50; extra == 'openai'
Description-Content-Type: text/markdown

# soul-schema 🧠

**Auto-generate semantic layers for any database using LLMs.**

[![PyPI version](https://badge.fury.io/py/soul-schema.svg)](https://badge.fury.io/py/soul-schema)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

> **Metadata only. No row data ever leaves your infrastructure.**

soul-schema connects to your database, reads the schema, samples a few rows for context, and uses an LLM to generate human-readable descriptions for every table and column. Corrections are remembered permanently — the semantic layer gets smarter over time.

---

## Why soul-schema?

| | Alation / Collibra | Unity Catalog | soul-schema |
|---|---|---|---|
| Cost | $100K+/yr | Databricks only | Free / OSS |
| Setup | Weeks | Days | Minutes |
| Metadata generation | Manual | Semi-auto | Automatic |
| Learns from corrections | ❌ | ❌ | ✅ |
| Works with any database | ✅ | ❌ | ✅ |
| Air-gapped (local LLM) | ❌ | ❌ | ✅ |

---

## Install

```bash
pip install soul-schema

# With Anthropic support
pip install soul-schema[anthropic]

# With OpenAI support  
pip install soul-schema[openai]

# Everything
pip install soul-schema[all]
```

---

## Quickstart

```bash
# Connect and generate descriptions
soul-schema connect \
  --db "postgresql://user:pass@localhost/mydb" \
  --llm anthropic \
  --key sk-ant-...

# Review and correct
soul-schema review

# Export to dbt
soul-schema export --format dbt

# Export to Vanna training data
soul-schema export --format vanna

# Check status
soul-schema status
```

---

## Works with any LLM

```bash
# Anthropic Claude (recommended)
soul-schema connect --db ... --llm anthropic --key sk-ant-...

# OpenAI
soul-schema connect --db ... --llm openai --key sk-...

# Ollama (fully local, air-gapped)
soul-schema connect --db ... --llm openai-compatible \
  --base-url http://localhost:11434/v1 \
  --model llama3.2

# Google Gemini
soul-schema connect --db ... --llm openai-compatible \
  --key AIza... \
  --base-url https://generativelanguage.googleapis.com/v1beta/openai \
  --model gemini-2.0-flash
```

---

## How it works

1. **Connect** — reads `INFORMATION_SCHEMA` for table/column metadata
2. **Sample** — fetches up to 10 rows per table to help the LLM understand context
3. **Generate** — LLM writes descriptions for every table and column
4. **Store** — saves to a human-readable markdown file (`schema_memory.md`)
5. **Learn** — corrections are locked and never overwritten by re-generation
6. **Export** — outputs dbt YAML, Vanna training data, or portable JSON

**Your data stays in your infrastructure.** soul-schema never stores or transmits row-level data. The memory file contains only metadata and LLM-generated descriptions.

---

## The memory file

soul-schema stores everything in a plain markdown file — `schema_memory.md`.

- Human-readable and editable by hand
- Version-controllable with git
- Locked columns (human corrections) are never overwritten
- Portable — bring it to any tool

---

## Export formats

```bash
soul-schema export --format json    # → soul_schema_export.json
soul-schema export --format dbt     # → schema.yml (dbt schema file)
soul-schema export --format vanna   # → Vanna AI training data
```

---

## Python API

```python
from soul_schema import SchemaConnector, SemanticGenerator, SchemaMemory

connector = SchemaConnector("postgresql://...")
generator = SemanticGenerator(provider="anthropic", api_key="sk-ant-...")
memory = SchemaMemory("./schema_memory.md")

schema = connector.get_full_schema()
for table, data in schema.items():
    result = generator.generate_table(
        table=table,
        columns=data["columns"],
        sample=data["sample"],
        row_count=data["row_count"],
    )
    memory.set_table(table, result["table_description"], result["columns"])

# Correct a description — locked forever
memory.correct("customers", "cust_ltv", "Customer lifetime value in USD")

# Export
print(memory.export_dbt())
```

---

## Supported databases

**Phase 1:** PostgreSQL, MySQL, SQLite  
**Coming soon:** BigQuery, Snowflake, Redshift, DuckDB, ClickHouse

---

## License

MIT — free for commercial use.

---

## Related projects

- [soul.py](https://github.com/menonpg/soul.py) — persistent memory for LLM agents
- [soul-agent](https://pypi.org/project/soul-agent/) — PyPI package with SoulMateClient
- [SoulMate](https://menonpg.github.io/soulmate/) — hosted memory API for enterprise agents
