Metadata-Version: 2.4
Name: learnml-sdk
Version: 0.1.2
Summary: Python SDK for the LearnML training data management platform
Home-page: https://github.com/milindjain0/learnml
Author: LearnML
Author-email: milindjain0@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: grpcio>=1.50.0
Requires-Dist: protobuf<6.0.0,>=5.26.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# LearnML — ML Data Management System

A unified platform for storing ML training data (traditional labeled data + LLM training data), model checkpoints, and training metrics.

**Stack:** C++17 · gRPC · Protocol Buffers · PostgreSQL · Grafana

---

## Quick Start

### Prerequisites

- Docker & Docker Compose
- CMake 3.20+
- C++17 compiler (GCC 9+ or Clang 10+)
- System packages: `grpc`, `protobuf`, `libpqxx`, `openssl`

### 1. Start PostgreSQL & Grafana

```bash
docker-compose up -d
```

This starts:
- **PostgreSQL 16** on `localhost:5432` (auto-runs migrations)
- **Grafana 10** on `http://localhost:3000` (admin/admin)

### 2. Build the Server

```bash
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
```

### 3. Run the Server

```bash
./build/learnml_server ../config.json
```

Server starts on `0.0.0.0:50051` by default.

### 4. Test with grpcurl

```bash
# Register a user
grpcurl -plaintext -d '{"email":"alice@test.com","password":"secret123","display_name":"Alice"}' \
  localhost:50051 learnml.AuthService/Register

# Login
grpcurl -plaintext -d '{"email":"alice@test.com","password":"secret123"}' \
  localhost:50051 learnml.AuthService/Login
# → Returns access_token and refresh_token

# Create a universe (use the access_token from login)
grpcurl -plaintext \
  -H "Authorization: Bearer <access_token>" \
  -d '{"name":"my-ml-project","description":"My first ML project"}' \
  localhost:50051 learnml.UniverseService/CreateUniverse

# Create a data point
grpcurl -plaintext \
  -H "Authorization: Bearer <token>" \
  -H "x-universe-id: <universe_id>" \
  -d '{"universe_id":"<id>","name":"sample_image","data_type":"IMAGE"}' \
  localhost:50051 learnml.DataService/CreateDataPoint

# Create a training run and log metrics
grpcurl -plaintext \
  -H "Authorization: Bearer <token>" \
  -d '{"universe_id":"<id>","name":"gpt2-finetune","model_name":"gpt2"}' \
  localhost:50051 learnml.MetricsService/CreateTrainingRun

grpcurl -plaintext \
  -H "Authorization: Bearer <token>" \
  -d '{"universe_id":"<id>","run_id":"<run_id>","snapshot":{"step":100,"metrics":{"loss":0.5,"perplexity":12.3}}}' \
  localhost:50051 learnml.MetricsService/LogMetrics
```

---

## Architecture

```
┌──────────────┐  ┌──────────┐  ┌─────────┐
│  CLI / SDK   │  │ React UI │  │ Grafana │
└──────┬───────┘  └────┬─────┘  └────┬────┘
       │               │             │ (PostgreSQL datasource)
       └───────┬───────┘             │
               ▼                     │
    ┌──────────────────┐             │
    │  Auth Interceptor│             │
    │  (JWT validation)│             │
    └────────┬─────────┘             │
             ▼                       │
    ┌──────────────────┐             │
    │  gRPC Services   │             │
    │  ┌─ Auth         │             │
    │  ├─ Universe     │             │
    │  ├─ Data         │             │
    │  ├─ Collection   │             │
    │  ├─ Checkpoint   │             │
    │  └─ Metrics      │             │
    └────────┬─────────┘             │
             ▼                       ▼
    ┌──────────────────┐  ┌──────────────────┐
    │   PostgreSQL     │  │  File System     │
    │   (metadata +    │  │  (blobs per      │
    │    RLS policies) │  │   universe)      │
    └──────────────────┘  └──────────────────┘
```

## Key Concepts

### Universe
The top-level isolation boundary. All data is scoped to a universe. Users are granted **READ**, **WRITE**, or **ADMIN** access.

### Data Points
Support traditional labeled data (image, video, PDF, text) and LLM training data (SFT, RL, pretraining).

### Collections
Logical groups of data points. A data point can belong to multiple collections (many-to-many).

### Checkpoints
Model checkpoint files with streaming upload/download APIs.

### Training Metrics
Logged per-step with batch or streaming APIs. Grafana queries PostgreSQL directly.

---

## Configuration

Edit `config.json` or use environment variables:

| JSON Key | Env Variable | Default | Description |
|----------|-------------|---------|-------------|
| `server_address` | `LEARNML_SERVER_ADDRESS` | `0.0.0.0:50051` | gRPC listen address |
| `db_host` | `LEARNML_DB_HOST` | `localhost` | PostgreSQL host |
| `db_port` | `LEARNML_DB_PORT` | `5432` | PostgreSQL port |
| `db_name` | `LEARNML_DB_NAME` | `learnml` | Database name |
| `jwt_secret` | `LEARNML_JWT_SECRET` | dev secret | JWT signing key |
| `storage_root` | `LEARNML_STORAGE_ROOT` | `/tmp/learnml_storage` | File storage root |

---

## Project Structure

```
learnml/
├── proto/learnml/          # Protobuf schemas (7 files)
├── src/
│   ├── auth/               # JWT, password hashing, gRPC interceptor
│   ├── db/                 # Connection pool, repository, migrations
│   ├── services/           # 6 gRPC service implementations
│   ├── server/             # gRPC server + main entry point
│   └── util/               # Config, file store
├── grafana/                # Dashboard JSON + provisioning
├── CMakeLists.txt
├── docker-compose.yml
├── config.json
└── README.md
```

## License

MIT
