Metadata-Version: 2.4
Name: sigil-lang
Version: 0.1.0
Summary: A programming language for LLM-to-LLM communication that transpiles to Python
Author: Eduard Ragea
License-Expression: MIT
Project-URL: Homepage, https://github.com/eduardragea/sigil
Project-URL: Repository, https://github.com/eduardragea/sigil
Project-URL: Issues, https://github.com/eduardragea/sigil/issues
Keywords: llm,transpiler,programming-language,token-efficiency,ai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Compilers
Classifier: Topic :: Software Development :: Code Generators
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tree-sitter>=0.20
Requires-Dist: tiktoken>=0.5
Requires-Dist: tomli>=2.0; python_version < "3.11"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: pydantic>=2.0; extra == "dev"
Requires-Dist: pygls>=1.3; extra == "dev"
Requires-Dist: lsprotocol>=2024.0; extra == "dev"
Provides-Extra: api
Requires-Dist: anthropic>=0.50; extra == "api"
Requires-Dist: openai>=1.0; extra == "api"
Provides-Extra: lsp
Requires-Dist: pygls>=1.3; extra == "lsp"
Requires-Dist: lsprotocol>=2024.0; extra == "lsp"
Dynamic: license-file

<p align="center">
  <h1 align="center">SIGIL</h1>
  <p align="center"><em>The programming language machines deserve.</em></p>
</p>

<p align="center">
  <a href="https://github.com/eduardragea/sigil/actions/workflows/ci.yml"><img src="https://github.com/eduardragea/sigil/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10+-blue.svg" alt="Python 3.10+"></a>
  <img src="https://img.shields.io/badge/tests-2700%2B%20passing-brightgreen.svg" alt="Tests">
  <img src="https://img.shields.io/badge/templates-100-purple.svg" alt="Templates">
  <img src="https://img.shields.io/badge/targets-Python%20%7C%20JS%20%7C%20TS%20%7C%20Go-orange.svg" alt="Targets">
  <a href="https://discord.gg/bQ4JrVkP"><img src="https://img.shields.io/badge/Discord-Join%20Server-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
</p>

<p align="center">
  <strong>91% fewer output tokens</strong> on real apps · <strong>86% faster</strong> generation · <strong>85% cheaper</strong>
</p>

<p align="center">
  <a href="https://sigil-pied.vercel.app">Website</a> ·
  <a href="https://sigil-pied.vercel.app/playground">Playground</a> ·
  <a href="https://discord.gg/bQ4JrVkP">Discord</a> ·
  <a href="https://github.com/eduardragea/sigil/blob/main/spec/sigil-spec.md">Spec</a>
</p>

---

> **Status:** Alpha — compiler, CLI, and VS Code extension functional. API may change.

## What is Sigil?

Sigil is a programming language designed for **LLM-to-LLM communication**. It is not meant for humans to read or write. It is the language AI agents use when generating, sharing, and executing code — optimised for **token density**, not human readability.

Every mainstream language was designed for humans. That made sense when humans wrote and read code. It doesn't make sense when machines are doing both. Sigil strips out everything machines don't need — verbose keywords, whitespace formatting, comments, syntactic sugar — and keeps pure computational intent.

```
Python (14 tokens):
def calculate_average(numbers: list[float]) -> float:
    """Calculate the arithmetic mean."""
    return sum(numbers) / len(numbers)

Sigil (5 tokens):
@avg [f]>f = +$/# $
```

Sigil **transpiles** to Python, Rust, Go, or TypeScript for execution. Think of it as a hyper-efficient IR (intermediate representation) for AI-generated code.

## Measured Token Reduction

Complete e-commerce app (product catalog, cart, checkout, dark theme UI) generated in one shot:

| Metric | Python | Sigil | Savings |
|--------|--------|-------|---------|
| **Output tokens** | 6,677 | 617 | **91% fewer** |
| **Lines of code** | 695 | 30 | **96% fewer** |
| **Generation time** | 66.4s | 9.4s | **86% faster** |
| **Cost** | $0.10 | $0.015 | **85% cheaper** |

The app runs on localhost with search, cart, checkout, and order confirmation.

Functional tasks (23 tasks, 100% accuracy):

| Strategy | Python tokens | Sigil tokens | Reduction |
|----------|-------------|-------------|-----------|
| Direct | 1,037 | 474 | **54%** |
| Dense (compact prompt) | 1,037 | 416 | **60%** |
| Compact (post-processed) | 1,037 | 418 | **60%** |

## Why?

- **Token tax:** Every token spent on human-readability features is a token an LLM doesn't need. At scale (millions of agent interactions/day), this is a real cost.
- **Context windows:** Density means more working space in finite context windows, where attention degrades with length.
- **Capability, not just cost:** Even with cheap inference, fitting more program state into context enables capabilities that raw token savings alone do not.

## Architecture

```
LLM reasons about task
    -> generates Sigil
        -> Sigil transpiles to target language
            -> executes
                -> structured output returns to LLM
```

**Current target:** Sigil -> Python (via Python ast module)

**Future targets:** Sigil -> Rust, Go, TypeScript, WASM, bytecode

## Project Structure

```
sigil/
├── grammar/            # Tree-sitter grammar definition
│   └── grammar.js      # Formal BNF-equivalent grammar
├── transpiler/         # Sigil -> Python transpiler
│   ├── parser.py       # Tokeniser + parser (AST generation)
│   ├── ir.py           # Intermediate representation
│   ├── codegen.py      # Python code generation from IR
│   └── cli.py          # Command-line interface
├── benchmarks/         # Token economy benchmarks
│   ├── humaneval/      # HumanEval benchmark translations
│   ├── mbpp/           # MBPP benchmark translations
│   └── results/        # Benchmark results and analysis
├── validation/         # LLM generation accuracy tests
│   ├── test_generate.py    # Can LLMs generate valid Sigil?
│   ├── test_roundtrip.py   # Sigil -> Python -> Sigil fidelity
│   └── test_accuracy.py    # Comparison vs Python generation
├── spec/               # Language specification
│   └── sigil-spec.md   # Formal language spec (BNF + semantics)
├── examples/           # Example Sigil programs
└── DESIGN_LOG.md       # Dated design decisions
```

## Installation

```bash
pip install sigil-lang
```

For development (includes pytest and black):

```bash
pip install sigil-lang[dev]
```

For LLM API integration (includes anthropic and openai SDKs):

```bash
pip install sigil-lang[api]
```

## Quick Start

```bash
# Transpile a .sigil file to Python
sigil transpile examples/hello.sigil --target python

# Transpile and execute
sigil run examples/hello.sigil

# Parse and type-check without executing
sigil check examples/hello.sigil

# Format a .sigil file
sigil fmt examples/hello.sigil

# Count tokens (compare Sigil vs generated Python)
sigil tokens examples/hello.sigil --compare
```

### From Source

```bash
git clone git@github.com:eduardragea/sigil.git
cd sigil

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Run tests
python -m pytest tests/
```

## Gradual Adoption: Inline Sigil in Python

You don't need to rewrite your codebase. Embed Sigil snippets directly in
existing Python files and migrate function-by-function.

```python
from sigil.inline import transpile, sigil_func

# Option 1: transpile a Sigil string and exec it
exec(transpile("@avg l = +l/#l"))
print(avg([1, 2, 3]))  # 2.0

# Option 2: replace a function stub with a Sigil implementation
@sigil_func("@greet n:s>s = f'Hello {n}'")
def greet(n): ...

print(greet("world"))  # Hello world
```

Both approaches produce standard Python at import time -- no runtime
overhead and full compatibility with type checkers and debuggers.

## Language Reference (Draft)

| Concept | Python | Sigil |
|---------|--------|-------|
| Variable binding | `x = 42` | `x:42` |
| Function definition | `def f(x): return x+1` | `@f x = x+1` |
| Conditional | `if x > 0: a else: b` | `x>0 ? a : b` |
| List comprehension | `[x**2 for x in xs if x>0]` | `xs\|>0\|^2` |
| Map + Filter | `map(fn, filter(p, xs))` | `xs\|p\|>fn` |
| Error handling | `try: ... except: ...` | `!{...} ~> {...}` |
| Type annotation | `x: int = 5` | `x:i=5` |
| Module import | `from math import sqrt` | `<math.sqrt>` |
| Async operation | `await fetch(url)` | `~>fetch url` |
| Pattern match | `match x: case 1: ...` | `x~{1:... _:...}` |

## Docker

```bash
# Build the image
docker build -t sigil-lang .

# Run Sigil CLI via Docker
docker run --rm -v $(pwd):/work sigil-lang transpile /work/app.sigil

# Run playground (API + web)
docker compose up
```

The playground exposes the API on `http://localhost:8000` and the web UI on `http://localhost:3000`.

## Roadmap

### Done

- [x] Formal grammar (43+ rules, Tree-sitter parser)
- [x] Sigil → Python transpiler with type inference
- [x] CLI (`transpile`, `run`, `check`, `fmt`, `tokens`, `init`)
- [x] VS Code extension, playground web app, REPL
- [x] AI tool integrations (Claude Code MCP, Cursor, Copilot, Codex, Grok)
- [x] JS/TS/Go transpilation targets, 100 codebook templates, diff protocol

### Phase 1: Trust the Transpiler ✅

- [x] **100% e2e accuracy** — 23/23 functional tasks pass with zero fine-tuning
- [x] **Codegen failures 35 → 13** — pipe map, fold lambda, filter, imports fixed
- [x] **Python-style calls** — `range(1, n+1)`, `s.replace(" ", "")` work natively
- [x] **Raw Python escape** — `%%{complex_python}%%` for patterns Sigil can't express
- [x] **Negative slices** — `xs[::-1]` works correctly

### Phase 2: Break the 60-80% Barrier

Current: 29% on apps (47% of Sigil is `%%{...}%%` escape hatches). Theoretical: 63% without escapes.

- [ ] **Fix dict literal parsing** — `json.dumps({"key": v})` in all positions (#281)
- [ ] **Multi-function file parsing** — external scanner for body termination (#284)
- [ ] **Template system** — `REST_CRUD "tasks" Task {fields}` → 50 lines Python (#282)
- [ ] **Full-stack app** — frontend + backend both in Sigil (#283)

### Phase 3: Launch

- [ ] **Publish v0.1.0 to PyPI** (#147)
- [ ] **Publish VS Code extension** (#182)
- [ ] **Write launch post** (#149)

### Phase 4: Sigil as a Reasoning Language

- [x] **Sigil-Plan** — agent planning notation (#216)
- [x] **Sigil Agent** — coding agent that thinks in Sigil (#218)
- [x] **Sigil-Think** — compressed chain-of-thought (#217)

### Phase 5: BaseLLM — Token-Native Encoding

Every number system is optimized for its consumer. Base2 for circuits. Base10 for humans. **BaseLLM for transformers.**

- [ ] **Layer 1: Custom tokenizer** — Sigil operators as single tokens, +30-50% compression (#293)
- [ ] **Layer 2: Semantic code tokens** — one token = one programming pattern (#294)
- [ ] **Layer 3: Semantic reasoning tokens** — one token = one reasoning primitive (#295)
- [ ] **Target: 80-90% fewer tokens** for complete agent sessions (code + reasoning + planning)

## AI Tool Integrations

Sigil works with every major AI coding tool:

| Tool | Setup | What it does |
|------|-------|-------------|
| **Claude Code** | MCP server + skills | 4 tools: parse, transpile, validate, count tokens |
| **Cursor** | `.cursor/rules/sigil.mdc` | AI generates correct Sigil in .sigil files |
| **GitHub Copilot** | `.github/copilot-instructions.md` | Copilot Chat understands Sigil syntax |
| **OpenAI Codex** | `AGENTS.md` | Codex reads project context automatically |
| **Grok** | FastAPI tool server | Function calling for parse/transpile/tokens |

See `integrations/` for setup instructions.

## Constrained Decoding (GBNF)

Force LLMs to generate **only valid Sigil syntax** using grammar-constrained decoding. This eliminates syntax errors entirely -- the model cannot produce tokens that violate the grammar.

Two grammar files are provided in `spec/`:

| File | Format | Used by |
|------|--------|---------|
| `spec/sigil.gbnf` | GBNF (GGML BNF) | llama.cpp, llama-cpp-python, text-generation-webui |
| `spec/sigil.ebnf` | Standard EBNF (ISO 14977) | Outlines, guidance, lm-format-enforcer |

### llama.cpp

Pass the grammar file with `--grammar-file`:

```bash
llama-cli \
  -m model.gguf \
  -p "Write a Sigil function that reverses a list:" \
  --grammar-file spec/sigil.gbnf
```

Or load the grammar string in llama-cpp-python:

```python
from llama_cpp import Llama

llm = Llama(model_path="model.gguf")

with open("spec/sigil.gbnf") as f:
    grammar_text = f.read()

output = llm(
    "Write a Sigil function that reverses a list:",
    grammar=LlamaGrammar.from_string(grammar_text),
)
```

### Outlines (vLLM / transformers)

Outlines accepts EBNF grammars for structured generation:

```python
import outlines

model = outlines.models.transformers("model-name")
generator = outlines.generate.cfg(model, open("spec/sigil.ebnf").read())

result = generator("Write a Sigil function that computes factorial:")
```

### guidance

```python
from guidance import models, gen

lm = models.LlamaCpp("model.gguf")

with open("spec/sigil.gbnf") as f:
    grammar = f.read()

lm += "Generate Sigil:\n" + gen(name="code", grammar=grammar)
```

## The Three-Layer Vision

Sigil is evolving from code compression into a full reasoning language for AI agents:

```
Layer 1: Sigil-Code   — compress Python syntax          (46% savings)  ← SHIPPED
Layer 2: Sigil-Plan   — compress agent planning          (70-80% est.) ← NEXT
Layer 3: Sigil-Think  — compress chain-of-thought        (70-80% est.) ← RESEARCH
```

**Why this matters:** An agent that thinks in Sigil-Plan and generates Sigil-Code has 2x the effective context window of a Python-native agent at the same API cost. This isn't just cheaper — it enables capabilities that don't fit in a Python agent's context.

```
Sigil-Plan example:
  P:add-error-handling target:f
    R:file → S:find @f → C:has !{}~>{}
    ?!C → E:wrap !{body}~>{fallback} → T:pytest → G:commit

vs natural language (4x more tokens):
  "I need to: 1) Read the file, 2) Find the function, 3) Check if it
   handles errors, 4) If not, add try/except, 5) Run tests, 6) Commit"
```

## Contributing

Sigil is in active development. If you're interested in language design, compiler engineering, or LLM optimisation, open an issue or reach out.

---

<p align="center">
  <em>Sigil is the lingua franca of machine intelligence.</em>
</p>
