Metadata-Version: 2.4
Name: concentriq-compute-sdk
Version: 0.1.0
Summary: Python SDK for the Concentriq Compute Platform - submit and manage distributed compute jobs
Project-URL: Homepage, https://proscia.com
Project-URL: Documentation, https://github.com/Proscia/concentriq-compute-sdk
Project-URL: Repository, https://github.com/Proscia/concentriq-compute-sdk
Project-URL: Changelog, https://github.com/Proscia/concentriq-compute-sdk/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/Proscia/concentriq-compute-sdk/issues
Author-email: Kyriakos Toulgaridis <kyriakos.toulgaridis@proscia.com>, Proscia TechOps <techops@proscia.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,compute,concentriq,distributed,jobs,kubernetes,ml,proscia
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: kubernetes<35.0.0,>=33.0.0
Requires-Dist: opentelemetry-api<2.0.0,>=1.39.0
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.39.0
Requires-Dist: opentelemetry-sdk<2.0.0,>=1.39.0
Requires-Dist: pydantic<3.0.0,>=2.12.0
Requires-Dist: pyyaml<7.0.0,>=6.0.0
Requires-Dist: requests<3.0.0,>=2.32.5
Requires-Dist: urllib3<3.0.0,>=2.6.1
Provides-Extra: dev
Requires-Dist: mypy>=1.9.0; extra == 'dev'
Requires-Dist: opentelemetry-sdk<2.0.0,>=1.39.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.12.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: types-requests; extra == 'dev'
Description-Content-Type: text/markdown

# Concentriq Compute SDK

Python SDK for the Concentriq Compute Platform - submit and manage distributed compute jobs with ease.

![Python 3.10+](https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12%20|%203.13-blue)
[![codecov](https://codecov.io/gh/Proscia/concentriq-compute-sdk/graph/badge.svg)](https://codecov.io/gh/Proscia/concentriq-compute-sdk)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

## Installation

```bash
pip install concentriq-compute-sdk
```

Or with uv:

```bash
uv add concentriq-compute-sdk
```

## Quick Start

### Submitting Jobs

```python
from concentriq_compute_sdk import ComputeClient, JobSpec, ResourceSpec

# Initialize client (auto-discovers Kubernetes config)
client = ComputeClient()

# Create job specification
job_spec = JobSpec(
    job_data={"input": "your-data"},
    resources=ResourceSpec(
        cpu="2000m",      # 2 CPU cores
        memory="4Gi",     # 4GB RAM
        gpu_count=1       # Optional: 1 GPU
    ),
    customer_image="your-org/processor:latest",
    timeout_seconds=3600,
    priority="high"
)

# Submit job
job_id = client.submit_job(job_spec)
print(f"Job submitted: {job_id}")
```

### NodePool Targeting (Advanced)

Control where your jobs run by specifying architecture, capacity type, storage, and GPU requirements:

```python
from concentriq_compute_sdk import ComputeClient, JobSpec, ResourceSpec, NodePoolSpec, GPUVRAM

# Request by VRAM tier (flexible, cost-efficient)
job_spec = JobSpec(
    job_data={"model": "llama-13b"},
    resources=ResourceSpec(cpu="8", memory="32Gi", gpu_count=1),
    nodepool=NodePoolSpec(
        gpu_memory_min_mib=GPUVRAM.STANDARD  # Gets A10G or L4 (22GB)
    )
)

# Request specific GPU model
job_spec = JobSpec(
    job_data={"model": "llama-70b-fp8"},
    resources=ResourceSpec(cpu="16", memory="64Gi", gpu_count=1),
    nodepool=NodePoolSpec(
        capacity_type="on-demand",  # Spot (default) or on-demand
        gpu_model="l40s"            # Need L40S for fp8 precision
    )
)

# ARM workload with large storage
job_spec = JobSpec(
    job_data={"task": "cpu-intensive"},
    resources=ResourceSpec(cpu="8", memory="16Gi"),
    nodepool=NodePoolSpec(
        architecture="arm64",       # amd64 (default) or arm64
        storage_tier="large"        # default (200GB), large (320GB), xlarge (500GB)
    )
)
```

**GPU VRAM Reference:**
- `GPUVRAM.STANDARD` (22,888 MiB / ~22GB): A10G or L4
- `GPUVRAM.LARGE` (45,776 MiB / ~45GB): L40S

**Defaults** (if nodepool not specified): `amd64` + `spot` + `default storage`

### Building a Controller

Controllers poll for work and submit jobs. You own the control flow.

```python
import time
from concentriq_compute_sdk import BaseController, JobSpec, ResourceSpec

class MyController(BaseController):
    def run(self):
        while True:
            # Poll your API for work
            work_items = self.fetch_work_from_api()

            if work_items:
                # Create job specs
                job_specs = [
                    JobSpec(
                        job_data=item,
                        resources=ResourceSpec(cpu="1000m", memory="2Gi")
                    )
                    for item in work_items
                ]

                # Submit with automatic logging/tracing
                job_ids = self.submit_jobs(job_specs)
                print(f"Submitted {len(job_ids)} jobs")

            time.sleep(60)

    def fetch_work_from_api(self):
        # Your logic here
        return []

# Run controller
controller = MyController()
controller.run()
```

### Building a Processor

Processors execute jobs. You own the execution flow.

```python
from concentriq_compute_sdk import BaseProcessor

class MyProcessor(BaseProcessor):
    def run(self):
        try:
            # Load job data (automatic from ConfigMap)
            job_data = self.load_job_data()
            self.on_job_started(self.job_id, job_data)

            # Your processing logic
            results = self.process(job_data)

            # Report completion
            self.report_complete(results)
            self.on_job_completed(self.job_id, results)

        except Exception as e:
            self.on_job_error(self.job_id, e)
            raise

    def process(self, job_data):
        # Your business logic here
        input_data = job_data.get("input")

        # Process...

        return {"status": "completed", "output": "result"}

# Run processor
processor = MyProcessor()
processor.run()
```

## Features

- **Simple API**: Only 3 methods to implement (controller) or 1 method (processor)
- **Type-Safe**: Pydantic models with validation
- **Versioned**: `v1` API for backward compatibility
- **Observable**: Built-in OpenTelemetry tracing and structured logging
- **Flexible Resources**: Specify exact CPU/memory/GPU per job
- **Automatic Cleanup**: Jobs and ConfigMaps cleaned up via TTL and owner references

## Architecture

**Job Flow:**
1. Your Controller runs in a loop, creating JobSpecs from your data source
2. ComputeClient submits Kubernetes Job with ConfigMap containing job data
3. Kueue schedules job when resources available
4. Your Processor loads job data from ConfigMap mount and executes
5. Your Processor reports results to your chosen destination

## Versioning

The SDK uses semantic versioning with a versioned API following Kubernetes API evolution:

```python
# Explicit v1alpha1 import (recommended for libraries)
from concentriq_compute_sdk.v1alpha1 import ComputeClient, JobSpec

# Convenience import (maps to current API, currently v1alpha1)
from concentriq_compute_sdk import ComputeClient, JobSpec
```

**Alpha API (v1alpha1)**: This is the first implementation and may have breaking changes in future releases. When the API stabilizes after production use:
- Graduate to `v1beta1` (feature-complete, needs validation)
- Then to `v1` (battle-tested, backward compatible)
- Keep `v1alpha1` for experimental features

## Requirements

- Python 3.10+
- Kubernetes cluster with Kueue installed
- Valid kubeconfig or in-cluster configuration

## Development

```bash
# Clone repository
git clone https://github.com/Proscia/concentriq-compute-sdk
cd concentriq-compute-sdk

# Install with dev dependencies
uv sync --all-extras

# Run tests
uv run pytest

# Lint
uv run ruff check src tests

# Type check
uv run mypy src
```

## Environment Variables

### ComputeClient Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `NAMESPACE` | `default` | Kubernetes namespace for jobs |
| `QUEUE_NAME` | `customer-queue` | Kueue queue name |
| `JOB_IMAGE` | `customer-job-processor:latest` | Default processor image |
| `SERVICE_ACCOUNT` | `default` | Kubernetes service account |
| `JOB_TTL_SECONDS` | `3600` | Job cleanup TTL (1 hour) |
| `LOG_LEVEL` | `INFO` | Logging level |

### Processor Configuration

| Variable | Description |
|----------|-------------|
| `JOB_ID` | Auto-set by platform |
| `CONFIG_MAP_NAME` | Auto-set by platform |

## License

Apache 2.0 - See [LICENSE](LICENSE) for details.

## Support

- Documentation: https://github.com/Proscia/concentriq-compute-sdk
- Issues: https://github.com/Proscia/concentriq-compute-sdk/issues
- Changelog: [CHANGELOG.md](CHANGELOG.md)
