Metadata-Version: 2.4
Name: cist-kernel
Version: 0.0.68
Summary: CLI for cist kernel telemetry and compilation
Author-email: tooig research lab <oyebamijo@tooig.com>
License: tooig software license
Requires-Python: >=3.9
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.9.0
Provides-Extra: cloud
Requires-Dist: fastapi>=0.115.0; extra == 'cloud'
Requires-Dist: modal>=0.74.0; extra == 'cloud'
Requires-Dist: uvicorn>=0.30.0; extra == 'cloud'
Provides-Extra: dev
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: build>=0.10.0; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# CIST Monorepo

Polyglot monorepo for the Contract Instruction Set (`.cis`) language.

This repository ships three user-facing pieces together:

- `cist-kernel` Python package — the CLI, the compiler, and the cloud compilation client.
- A VS Code extension bundled inside the package — `cist init` installs it into your local VS Code extensions folder. Recognises both `.cis` source files and `.mex` compiled binaries.
- A build orchestrator (`scripts/build.py`) for cutting releases.

## Language Snapshot

```cis
use math
use compute
use risk

market_data := from data (source: "binance", symbols: ["BTC-USD"], resolution: "1m")

sma_20 := math.sma(market_data.close, 20)
signal := compute.crossover(market_data.price, sma_20)

if signal and position.size == 0:
    execute(market_buy(size: risk.calculate_size(portfolio_value: 10000)))
elif position.pnl < -100:
    execute(close_position())
```

## Repository Layout

```text
cist/
├── pyproject.toml
├── cist.toml                   ← workspace config (created by cist init)
├── README.md
├── .gitignore                  ← *.mex and .env excluded by default
├── scripts/
│   └── build.py
├── cist_cli/
│   ├── __init__.py
│   ├── main.py
│   ├── network.py
│   ├── compiler/               ← NEW: compilation pipeline
│   │   ├── __init__.py
│   │   ├── lexer.py            ← CISTCompiler: syntax + semantic validation → IR
│   │   ├── bundle.py           ← compile bundle + harness generation
│   │   └── mex.py              ← MexWriter / MexReader: .mex binary format
│   ├── cloud/
│   │   ├── service.py          ← local /compile + /telemetry service
│   │   └── modal_stub.py       ← optional Modal deployment entrypoint
│   ├── workspace/              ← NEW: workspace scaffold
│   │   ├── __init__.py
│   │   ├── init.py             ← init_workspace(): creates directory structure
│   │   └── templates.py        ← all generated-file templates
│   └── assets/
│       └── *.vsix              ← bundled VS Code extension
├── tests/
│   ├── test_compiler.py
│   ├── test_mex.py
│   ├── test_workspace.py
│   ├── test_entrypoint.py
│   ├── test_build_script.py
│   └── test_vscode_grammar.py
└── extensions/
    ├── package.json            ← declares .cis and .mex language contributions
    ├── language-configuration.json
    └── syntaxes/
        └── cis.tmLanguage.json
```

Each contract lives in its own sub-directory under `contracts/`:

```text
contracts/
└── {contract_name}/
    ├── {contract_name}.cis     ← main source file
    ├── program.mex             ← compiled model-executable bundle
    └── bench/
        ├── example.cis         ← test/bench scenarios (extend freely)
        └── manifest.md         ← safe-data instructions for the bench harness
```

## Getting Started

### 1. Install or upgrade

```bash
python -m pip install --upgrade cist-kernel
```

The `--upgrade` flag matters. A plain `pip install cist-kernel` keeps an existing older version and will not pull in new features.

If `cist` is not on your `PATH` after install, use the fallback:

```bash
python -m cist_cli
```

On Windows, the installer will print the exact `PowerShell` one-liner to add the scripts folder to your `PATH` permanently the first time you run via `python -m cist_cli`.

### 2. Initialise a workspace (if you haven't already)

```bash
cist init              # in the current directory
cist init --path ~/my-workspace
```

`cist init` creates the following structure from scratch (safe to re-run — existing files are never overwritten):

```text
<workspace>/
├── cist.toml                    project name, kernel version, compiler settings
├── .env                         CIST_CLOUD_API_KEY placeholder
├── .gitignore                   *.mex and .env pre-seeded
└── contracts/
    └── example/
        ├── example.cis          starter contract (edit freely)
        └── bench/
            ├── example.cis      bench / test scenarios (extend freely)
            └── manifest.md      safe-data instructions for the bench harness
```

It also silently extracts the bundled VS Code extension into every detected editor extensions folder (`.vscode`, `.vscode-insiders`, `.antigravity`), removing any stale legacy versions. Reload the window once with `Ctrl+Shift+P → Developer: Reload Window` to activate syntax highlighting for `.cis` and `.mex` files.

### 3. Add your API key

Open the `.env` file and fill in your key:

```bash
CIST_CLOUD_API_KEY=sk_live_your_key_here
```

### 4. Write your contract

Edit `contracts/example/example.cis` or create a new sub-directory:

```cis
use math
use hyper

contract OilPriceMonitor:
    which defines:
        while True:
            oil_price = hyper.get_price("OIL")

            then:
                if oil_price > 80.0:
                    print("Consider selling.")
```

Add test scenarios in `contracts/example/bench/`:

```cis
contract BenchHighPrice:
    which defines:
        oil_price = 90.0
        then:
            if oil_price > 80.0:
                print("bench: high price path OK")
```

### 5. Compile

```bash
cist compile contracts/example/example.cis
```

If you only want local validation plus a local `.mex` artifact, use the build command instead of the cloud stream:

```bash
cist build contracts/example/example.cis
```

What happens:

1. **Syntax check** — the compiler validates brackets, strings, `use` imports, `from data(…)` expressions, selective imports, block markers, indentation, and scans `contract` and `write fn` declarations.
2. **Bundle emit** — if the file is valid, `program.mex` is written next to the source file. This binary now stores the model-facing compile bundle: source, IR, bench manifests, harness instructions, and the generated task input.
3. **Cloud stream** — the `.mex` bundle is streamed to the CIST Cloud compiler. The cloud worker materialises or refines Rust contracts, runs the Composer bundler, and streams logs, artifacts, and errors back in real time.

```text
✓ example.cis  42 lines · modules: math, hyper · 0 data stream(s)
✓ Bundle written → contracts/example/program.mex
Connecting to CIST Cloud...
```

Add `--deploy` to mark the build for live execution:

```bash
cist compile contracts/example/example.cis --deploy
```

Add `--verbose` to see the full module and data-stream breakdown:

```bash
cist compile contracts/example/example.cis --verbose
```

---

## CLI Reference

### `cist init [--path <dir>]`

Initialises a CIST workspace.

- Creates `contracts/{name}/{name}.cis` and `contracts/{name}/bench/example.cis`.
- Creates `contracts/{name}/bench/manifest.md` so the safe layer has explicit data-generation instructions.
- Writes `cist.toml` (project name, kernel version, compiler settings).
- Writes `.env` stub with `CIST_CLOUD_API_KEY` placeholder.
- Seeds `.gitignore` with `*.mex` and `.env` (appends if the file already exists; never duplicates).
- Removes stale legacy extension versions and silently extracts the bundled `.cis` + `.mex` VS Code extension.

### `cist compile <file.cis> [--deploy] [--verbose]`

Compiles a `.cis` contract.

- **Step 1 — local validation**: bracket matching, quote closure, `use` imports, `from data(…)` expressions, mixed tab/space indentation, UTF-8 encoding.
- **Step 2 — bundle emit**: writes a `program.mex` binary alongside the source. The binary is zlib-compressed, integrity-checked with CRC-32, and contains the model-executable compile bundle.
- **Step 3 — cloud stream**: streams the compile bundle to the CIST Cloud. Logs, generated file artifacts, bench output, and errors are printed in real time.
- `--deploy` flags the build as live.
- `--verbose` prints the full module list and data-stream count.

### `cist build <file.cis> [--debug]`

Builds the local `.mex` artifact without calling the cloud compiler.

- Runs the same local validation path as `cist compile`.
- Writes `program.mex` next to the source file.
- `--debug` writes the `.mex` payload as readable JSON instead of the compressed binary container.

### `cist check <file.cis>`

Runs the validator only and prints any syntax or structural diagnostics without writing a `.mex` file.

### `cist serve [--host <addr>] [--port <port>]`

Runs the local cloud compiler service with two routes:

- `/compile` — accepts a compile bundle and streams SSE logs
- `/telemetry` — returns the latest compile or rig state as JSON

Install the optional cloud extra first if you want to run this locally:

```bash
python -m pip install "cist-kernel[cloud]"
```

### `cist telemetry [--session-id <id>]`

Fetches the latest compiler telemetry from the configured cloud endpoint.

### `cist sync-vscode-extension`

Re-installs or updates the bundled VS Code extension without running a full `init`.

---

## The `.mex` Binary Format

When `cist compile` succeeds it writes a `program.mex` file alongside the source. The format is:

| Offset | Size | Content |
| ------ | ---- | ------- |
| 0 | 4 B | Magic: `MEX\x00` |
| 4 | 2 B | Format version (big-endian uint16, currently `1`) |
| 6 | 4 B | CRC-32 of the compressed payload (big-endian uint32) |
| 10 | N B | zlib-compressed UTF-8 JSON payload |

The payload holds the model-executable compile bundle: source text, compiler IR, bench and manifest documents, nursery schema hints, harness instructions, and the model task input.

`.mex` files are recognised by the VS Code extension (file-tree icon, language ID `mex`) but are intentionally non-editable binary blobs. They are excluded from version control via the workspace `.gitignore`.

---

## Configuration

### `cist.toml`

Created by `cist init` at the workspace root. Controls project metadata and compiler behaviour:

```toml
[project]
name = "example"
version = "0.1.0"
description = ""

[kernel]
version = "0.0.38"   # kernel version this workspace was initialised with

[compiler]
strict = false        # set true to treat warnings as errors
debug = false         # set true to write program.mex as readable JSON during development

[cloud]
backend = "auto"     # auto | nineth | template
model = "1984-m3-0404"

[runtime]
composer_root = "composer"
telemetry_poll_seconds = 5
```

### Environment Variables

Loaded from `.env` (if present) and from the shell environment.

| Variable | Required | Default | Purpose |
| -------- | -------- | ------- | ------- |
| `CIST_CLOUD_API_KEY` | Yes (for compile) | — | Bearer token for the cloud compiler |
| `CIST_CLOUD_ENDPOINT` | No | `https://api.cist.io/v1` | Override the cloud endpoint |
| `CIST_CLOUD_MODEL` | No | `1984-m3-0404` | Model name used by the Nineth-backed compiler worker |
| `CIST_CLOUD_BACKEND` | No | `auto` | `auto`, `nineth`, or deterministic `template` fallback |
| `CIST_COMPOSER_ROOT` | No | — | Override the Composer runtime root on the cloud worker |
| `CIST_ENABLE_LIVE_RUN` | No | `0` | Set to `1` to let deploy builds continue from bench into `composer live` |
| `COMPOSER_EXECUTE_LIVE_TRADES` | No | `0` | Manual `composer live` runs only submit venue orders when this is `1`; the cloud service injects it for live deploy subprocesses |
| `CIST_SKIP_VSCODE_INSTALL` | No | — | Set to `1` to skip extension install (useful in CI) |
| `VSCODE_EXTENSIONS` | No | — | Override the VS Code extensions root directory |

---

## Validation Artifacts

This revision includes permanent run logs under [logs/2026-04-11](logs/2026-04-11):

- [logs/2026-04-11/python-pytest.log](logs/2026-04-11/python-pytest.log) — `46 passed`
- [logs/2026-04-11/composer-lifecycle.log](logs/2026-04-11/composer-lifecycle.log) — `22 passed, 0 failed, 3 ignored`
- [logs/2026-04-11/composer-bench.log](logs/2026-04-11/composer-bench.log)
- [logs/2026-04-11/composer-live.log](logs/2026-04-11/composer-live.log)
- [logs/2026-04-11/cist-init.log](logs/2026-04-11/cist-init.log)
- [logs/2026-04-11/cist-build.log](logs/2026-04-11/cist-build.log)

For the fuller architecture walkthroughs, see [report.md](report.md) and [composer/report.md](composer/report.md).

---

## Build Artifacts

Build both deliverables with:

```bash
python scripts/build.py
```

Expected outputs:

- `dist/cist_kernel-*.whl` — Python wheel containing the CLI and the bundled `.vsix`.

The build script verifies that the VSIX version inside the wheel matches the project version in `pyproject.toml` before declaring success.

---

## Notes

- Local validation is lightweight and non-executing. It never runs contract code.
- Cloud compilation requires `CIST_CLOUD_API_KEY` unless you are targeting a local anonymous endpoint such as `http://127.0.0.1:8000`.
- The VS Code grammar uses Python-style scope categories with CIST-specific handling for `use`, `contract`, `write fn`, `which defines:`, `then:`, and `from data(…)`.
- The `cist_cli/compiler/` and `cist_cli/workspace/` packages are independently importable for use in tooling or test fixtures.
