Metadata-Version: 2.4
Name: shardorm
Version: 0.0.4
Summary: A lightweight Python micro-ORM with sharding, replication, and failover layer
License: AGPL-3.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: psycopg[binary]>=3.0.0
Requires-Dist: psycopg_pool>=3.0.0

# ShardORM

> A lightweight Python micro-ORM with automatic PostgreSQL sharding, replication, failover, and schema synchronization.
>
> **Zero cluster management. Zero coordination layers. Pure Python over standard Postgres.**

ShardORM is built directly on top of **psycopg 3**, **psycopg_pool**, and the Python standard library. It automatically distributes data across PostgreSQL shards while keeping your schema synchronized on every node.

---

## Features

🗄️ Automatic PostgreSQL sharding
Deterministically distributes records across multiple PostgreSQL database shards without requiring manual routing logic in your application code.

🔄 Consistent hashing with virtual nodes
Uses a consistent hash ring with virtual nodes (CRC32), ensuring that adding or removing shards only requires migrating a minimal fraction of keys.

📦 Configurable replication factor
Allows flexible definition of how many shards a record should be replicated to for fault tolerance (e.g., replicating across 2 or 3 shards).

⚡ Automatic read failover
Queries target shards sequentially when the shard key is present, falling back automatically until the first responsive shard succeeds.

🌍 Full table replication (full_sync)
Enables complete synchronization of smaller tables (such as global settings or lookup data) across all configured shards.

🔀 Online shard rebalancing
Supports scaling and decommissioning shards (draining_shards) with live background data migration (rescale).

📜 SQL-based migrations
Keeps database schemas strictly synchronized across all shards—either via ad-hoc commands or versioned .sql migration files (acting like a mini-Alembic).

🔒 Optional distributed transactions (PostgreSQL 2PC)
Enables true distributed "all-or-nothing" writes across multiple shards using PostgreSQL's native Two-Phase-Commit (PREPARE TRANSACTION).

🏊 Built-in connection pooling
Leverages performant and robust per-shard connection management via psycopg_pool, including automatic timeouts and connection cleanup.

🛡️ Built-in SQL Injection Protection (strict identifier whitelisting)
Validates all table and column names strictly against a whitelist regular expression and safely masks them via psycopg.sql.Identifier to prevent structural injection.

🛑 Built-in Circuit Breaker (protection against cascading failures)
Detects failing shards in real time, immediately short-circuiting connection attempts during a cooldown period to prevent thread pool exhaustion.

⚡ Parallel scatter-gather queries (thread-pool powered fanout)
Executes global and multi-shard read operations concurrently using an optimized thread pool to keep response latencies minimal.

---

## Installation

```bash
pip install shardorm
```

Requirements:

- Python 3.11+
- PostgreSQL 13+
- psycopg 3
- psycopg_pool

---

## Configuration

ShardORM loads its configuration from:

```
./shardorm.config.json
```

or

```
$SHARDORM_CONFIG
```

Generate a template:

```bash
shardorm init-config
```

Example:

```json
{
  "replication_factor": 2,
  "min_write_quorum": 1,
  "shards": [
    "postgresql://user:password@localhost/db1",
    "postgresql://user:password@localhost/db2"
  ],
  "table_policies": {
    "users": {
      "mode": "sharded",
      "shard_key": "id"
    },
    "countries": {
      "mode": "full_sync"
    }
  }
}
```

---

## Table Policies

### Sharded

Rows are distributed across the cluster using a consistent hash ring.

```json
{
  "users": {
    "policy": "sharded",
    "shard_key": "id"
  }
}
```

### Full Sync

Rows are replicated to every configured shard.

```json
{
  "countries": {
    "policy": "full_sync"
  }
}
```

---

## CLI

```bash
shardorm init-config
shardorm status
shardorm make-migration create_users
shardorm migrate
shardorm add-table users id:UUID:PK email:TEXT
shardorm add-column users age:INTEGER
shardorm drop-table users --yes
shardorm rescale users
shardorm rescale users --apply
```

---

## FastAPI Example

```python
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from shardorm import ShardORM

app = FastAPI(title="ShardORM Example")

db = ShardORM.from_config()


class UserCreate(BaseModel):
    id: str
    name: str
    email: str


@app.on_event("shutdown")
def shutdown():
    db.close()


@app.get("/users")
def get_all_users():
    try:
        users = db.table("users").select_all_shards()
        return {
            "status": "success",
            "count": len(users),
            "users": users
        }
    except Exception as e:
        raise HTTPException(500, str(e))


@app.post("/users")
def create_user(user: UserCreate):
    try:
        result = (
            db.table("users")
            .insert(
                id=user.id,
                data=user.model_dump()
            )
        )

        return {
            "status": "success",
            "write_result": result
        }

    except Exception as e:
        raise HTTPException(500, str(e))


@app.get("/users/{user_id}")
def get_user(user_id: str):
    users = (
        db.table("users")
        .where(id=user_id)
        .select()
    )

    if not users:
        raise HTTPException(404, "User not found")

    return users[0]


@app.get("/cluster/status")
def cluster_status():
    return {
        "shards": db.status()
    }
```

---

## How It Works

```
Shard Key
     │
     ▼
Hash Function
     │
     ▼
Consistent Hash Ring
     │
     ▼
Virtual Nodes
     │
     ▼
Replication Factor
     │
     ▼
Destination Shards
```

Only the required shards are contacted for reads and writes. When new shards are added, only the affected rows are moved during rebalancing.

---

## Two-Phase Commit (Optional)

Enable atomic distributed transactions for a table:

```json
{
  "orders": {
    "policy": "sharded",
    "write_mode": "2pc"
  }
}
```

Internally this uses PostgreSQL's native:

```sql
PREPARE TRANSACTION
COMMIT PREPARED
ROLLBACK PREPARED
```

> PostgreSQL requires `max_prepared_transactions > 0`.

---

## Why ShardORM?

| Feature | ShardORM |
|----------|-----------|
| ORM | Lightweight |
| Pure SQL | ✅ |
| psycopg 3 | ✅ |
| Connection Pooling | ✅ |
| Sharding | ✅ |
| Replication | ✅ |
| Read Failover | ✅ |
| Online Rebalancing | ✅ |
| SQL Migrations | ✅ |
| PostgreSQL 2PC | ✅ |

---

## License

Licensed under the **GNU Affero General Public License v3.0 (AGPL-3.0)**.
