Metadata-Version: 2.4
Name: clear-repair
Version: 1.0.0
Summary: CLEAR: Closed-Loop Engine for Autonomous Code Repair
Author: Rohan Burman
Project-URL: Homepage, https://github.com/rrburman23/CLEAR
Project-URL: Documentation, https://github.com/rrburman23/CLEAR/tree/main/docs
Project-URL: Repository, https://github.com/rrburman23/CLEAR
Project-URL: Issues, https://github.com/rrburman23/CLEAR/issues
Keywords: llm,program-repair,code-repair,automated-program-repair,benchmarking,local-inference,ai
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: langchain>=0.2.0
Requires-Dist: langchain-ollama>=0.1.0
Requires-Dist: langgraph>=0.1.0
Requires-Dist: docker>=7.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pytest>=8.0.0
Requires-Dist: colorama>=0.4.6
Requires-Dist: pandas>=2.2.0
Requires-Dist: matplotlib>=3.8.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: typer>=0.9.0
Requires-Dist: rich>=14.0.0
Provides-Extra: dev
Requires-Dist: build>=1.2.0; extra == "dev"
Requires-Dist: ruff>=0.6.0; extra == "dev"

# CLEAR: Closed-Loop Engine for Autonomous Repair

**Author:** Rohan Burman
**Programme:** MSc Artificial Intelligence, Queen Mary University of London
**Year:** 2026
**Version:** 0.1.0

CLEAR is a local-first autonomous software-repair framework for repairing
faulty Python programs using language-model generation and deterministic,
test-driven verification.

A repair is counted as successful **only when the generated candidate passes
the complete supplied pytest oracle inside the CLEAR execution sandbox**.

CLEAR is distributed both as:

- an installable Python command-line application; and
- a complete source repository containing the benchmark corpus,
  experiment configuration, documentation and reproducibility artefacts.

---

## Overview

Conventional code generation is typically open-loop: a model generates source
code, but the generated implementation is not intrinsically guaranteed to be
functionally correct.

CLEAR instead implements a bounded closed-loop process:

1. a local language model proposes a complete Python implementation;
2. CLEAR normalises the generated candidate;
3. the candidate is executed against an immutable pytest oracle;
4. execution occurs inside an ephemeral Docker sandbox;
5. failed verification is converted into compact feedback;
6. the model may generate another candidate;
7. the loop terminates on verified success or a defined terminal condition.

The language model does not directly control arbitrary filesystem actions or
test execution. Candidate processing, sandbox execution, feedback handling,
state transitions and termination are controlled deterministically by CLEAR.

---

## Core Features

- **Closed-loop autonomous repair**
- **Local language-model inference through Ollama**
- **Deterministic pytest verification**
- **Ephemeral Docker execution sandbox**
- **Network-disabled candidate execution**
- **Bounded repair iterations and subprocess timeouts**
- **Repeated-candidate stagnation detection**
- **Structured terminal failure classification**
- **Standalone repair artefact export**
- **Multi-model benchmark evaluation**
- **Automated CSV, JSON, Markdown, LaTeX and graph reporting**
- **67-task three-tier benchmark corpus**
- **Installable `clear-repair` command-line application**
- **Runtime diagnostics and model availability inspection**

---

## System Architecture

```text
Faulty Python program
          +
Immutable pytest oracle
          │
          ▼
┌─────────────────────────┐
│ CLEAR Graph Orchestrator│
│                         │
│ deterministic state,    │
│ routing and termination │
└───────────┬─────────────┘
            │ repair context
            ▼
┌─────────────────────────┐
│ Local Language Model    │
│       (Ollama)          │
└───────────┬─────────────┘
            │ complete Python candidate
            ▼
┌─────────────────────────┐
│ Candidate Normalisation │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ Docker Sandbox + pytest │
└───────────┬─────────────┘
            │
       ┌────┴─────┐
       │          │
     PASS       FAILURE
       │          │
       ▼          └──────► compact feedback
Verified repair              │
                              └────► next model attempt
```

---

# Installation

## Runtime Requirements

CLEAR requires:

- Python 3.10 or later;
- Docker;
- Ollama.

A GPU is optional but strongly recommended for practical local-model
inference.

The CLEAR Python package does **not** bundle Docker, Ollama or language-model
weights.

---

## Install the Packaged CLI

Install a built CLEAR wheel:

```console
python -m pip install .\dist\clear_repair-1.0.0-py3-none-any.whl
```

Confirm the installation:

```console
clear-repair version
```

Example output:

```text
CLEAR 0.1.0
```

Check the runtime environment:

```console
clear-repair doctor
```

A healthy installation verifies:

- CLEAR package availability;
- Docker CLI availability;
- Docker daemon availability;
- `clear-executor` sandbox image;
- Ollama runtime availability;
- default model availability.

Example:

```text
       CLEAR Runtime Diagnostics
┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ Component      ┃ Status ┃ Details    ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ CLEAR          │ ✓ PASS │ Installed  │
│ Docker CLI     │ ✓ PASS │ Available  │
│ Docker daemon  │ ✓ PASS │ Running    │
│ clear-executor │ ✓ PASS │ Available  │
│ Ollama CLI     │ ✓ PASS │ Available  │
│ Ollama runtime │ ✓ PASS │ Running    │
│ Default model  │ ✓ PASS │ gemma4:12b │
└────────────────┴────────┴────────────┘

CLEAR is ready to run.
```

---

## Install from Source

For development, benchmark reproduction, modification or inspection of the
complete research artefact:

```console
git clone https://github.com/rrburman23/CLEAR.git
cd CLEAR

python -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install -e .
```

On Linux/macOS:

```console
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
```

Editable installation exposes the same `clear-repair` command while allowing
changes to the source tree to take effect during development.

---

# Docker Sandbox Setup

Build the CLEAR execution image from the repository root:

```console
docker build -t clear-executor .
```

Ensure Docker is running before executing a repair.

Verify the runtime with:

```console
clear-repair doctor
```

---

# Ollama Model Setup

The final default model is:

```text
gemma4:12b
```

Install it with:

```console
ollama pull gemma4:12b
```

Inspect CLEAR model availability with:

```console
clear-repair models
```

Example:

```text
                    CLEAR Model Status
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓
┃ Model               ┃ Configured ┃ Installed ┃ Default ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩
│ gemma4:12b          │     ✓      │     ✓     │    ★    │
│ deepseek-coder:6.7b │     ✓      │     ✓     │         │
│ llama3.1:8b         │     ✓      │     ✓     │         │
│ qwen2.5-coder:7b    │     ✓      │     ✓     │         │
│ gemma4:e4b          │     ✓      │     ✓     │         │
│ qwen2.5-coder:3b    │     ✓      │     ✓     │         │
│ granite-code:8b     │     ✓      │     ✓     │         │
│ ornith:9b           │     ✓      │     ✓     │         │
└─────────────────────┴────────────┴───────────┴─────────┘
```

To additionally display other models available through Ollama:

```console
clear-repair models --all
```

The presence of a model in Ollama does not automatically make it part of the
principal CLEAR benchmark experiment.

---

# Command-Line Application

The installed application exposes:

```console
clear-repair repair
clear-repair benchmark
clear-repair doctor
clear-repair models
clear-repair version
```

Show the top-level interface:

```console
clear-repair --help
```

---

## Repair a Python Program

```console
clear-repair repair `
    --code path/to/target.py `
    --test path/to/test_target.py
```

Example using the included factorial benchmark:

```console
clear-repair repair `
    --code tests/benchmarks/single_fault/logic/factorial/target.py `
    --test tests/benchmarks/single_fault/logic/factorial/test_logic.py
```

CLEAR generates candidate source, verifies it inside the sandbox and applies a
repair only when the supplied test suite passes.

---

## Save Standalone Repair Artefacts

```console
clear-repair repair `
    --code path/to/target.py `
    --test path/to/test_target.py `
    --save-run
```

A standalone run may retain:

```text
execution.log
original.py
repair.patch
repaired.py
result.json
test_suite.py
```

A custom output location can also be supplied:

```console
clear-repair repair `
    --code path/to/target.py `
    --test path/to/test_target.py `
    --save-run `
    --log-dir tests/manual_runs
```

---

## Run Benchmark Experiments

Run all configured principal models against the complete discovered corpus:

```console
clear-repair benchmark
```

Run one model:

```console
clear-repair benchmark `
    --models gemma4:12b
```

Run T1 only:

```console
clear-repair benchmark `
    --difficulties single_fault
```

Run T2 and T3:

```console
clear-repair benchmark `
    --difficulties intermediate advanced
```

Restrict evaluation to selected categories:

```console
clear-repair benchmark `
    --types logic syntax security
```

Combine model, difficulty and category filters:

```console
clear-repair benchmark `
    --models gemma4:12b `
    --difficulties single_fault `
    --types logic
```

Increase the per-benchmark subprocess timeout:

```console
clear-repair benchmark --timeout 600
```

The complete benchmark and research-reproduction workflow is intended to be
used with the source repository so that the benchmark corpus, manifest and
generated experiment artefacts remain available together.

---

# Python Module Interfaces

The original module interfaces remain supported.

Single repair:

```console
python -m src.main `
    --code path/to/target.py `
    --test path/to/test_target.py
```

Benchmark evaluation:

```console
python -m run_benchmarks
```

The installed `clear-repair` interface is recommended for normal use, while
the direct Python module interfaces remain useful for development,
debugging and reproducibility.

---

# Benchmark Corpus

The final CLEAR benchmark corpus contains **67 Python repair tasks** and
 **268 human-authored pytest tests**.

The corpus is divided into three difficulty tiers.

| Tier | Directory | Tasks | Definition |
| --- | --- | ---: | --- |
| T1 | `single_fault` | 51 | One intentionally seeded defect |
| T2 | `intermediate` | 10 | Multiple interacting defects within one broad fault category |
| T3 | `advanced` | 6 | Multiple interacting defects spanning different fault categories |

The benchmark manifest records **105 intentionally seeded faults** across the
complete corpus.

---

## Benchmark Categories

The T1 and T2 corpora contain tasks drawn from:

```text
algorithm
api
concurrency
data_structure
edge_case
exception
logic
oop
python
security
syntax
```

Advanced T3 tasks are reported under:

```text
mixed
```

---

## Benchmark Directory Layout

```console
tests/benchmarks/
├── benchmark_manifest.json
│
├── single_fault/
│   └── <category>/
│       └── <benchmark_name>/
│           ├── target.py
│           └── test_*.py
│
├── intermediate/
│   └── <category>/
│       └── <benchmark_name>/
│           ├── target.py
│           └── test_*.py
│
└── advanced/
    └── mixed/
        └── <benchmark_name>/
            ├── target.py
            └── test_advanced.py
```

Each benchmark contains:

- an intentionally faulty `target.py`;
- a deterministic pytest verification suite.

---

# Benchmark Validation

Before evaluation, CLEAR validates the discovered benchmark corpus.

The validation process checks that:

- each benchmark contains a target;
- each benchmark contains a discoverable pytest suite;
- test files contain executable pytest tests;
- deliberately malformed syntax targets can still be inspected safely;
- the original seeded target does not already satisfy the oracle.

The repository validator can also be run directly:

```console
python tests/validate_benchmarks.py
```

---

# Benchmark Manifest

The machine-readable corpus description is stored at:

```console
tests/benchmarks/benchmark_manifest.json
```

It records metadata including:

- benchmark identifier;
- difficulty tier;
- fault category;
- benchmark name;
- target path;
- test-suite path;
- pytest test count;
- seeded fault descriptions;
- intended fault count.

The manifest provides a reproducible description of the final 67-task corpus.

---

# Principal Experiment Models

The final dissertation comparison evaluates eight principal models:

| Model | Experimental role |
| --- | --- |
| `gemma4:12b` | Larger Gemma comparator and final default |
| `deepseek-coder:6.7b` | Code-specialised comparator |
| `llama3.1:8b` | General-purpose comparator |
| `qwen2.5-coder:7b` | Code-specialised comparator |
| `gemma4:e4b` | Efficiency-oriented Gemma comparator |
| `qwen2.5-coder:3b` | Compact code-specialised comparator |
| `granite-code:8b` | Code-specialised comparator |
| `ornith:9b` | Agentic-coding-oriented comparator |

All principal models use the same external CLEAR contract:

```text
faulty source
+ immutable oracle
+ optional verification feedback
        │
        ▼
complete Python source
```

The principal model profiles use:

- `code_only` output protocol;
- temperature `0.0`;
- 2048-token generation budget;
- deterministic CLEAR-controlled action construction.

Model-specific response normalisation may remove reasoning wrappers while
preserving the same external protocol.

---

# Optional Experimental Models

CLEAR may retain separately configured experimental models, including:

```text
deepseek-r1:8b
smallthinker:3b
```

These models are not part of the final eight-model principal evaluation and
are excluded from the default benchmark model list.

They may be used for explicit exploratory experiments where their differing
reasoning or generation requirements are documented separately.

---

# Evaluation Metrics

CLEAR reports six primary metrics.

### Success Rate (SR)

Percentage of attempted benchmark cases repaired successfully.

### Pass@1

Percentage of attempted cases repaired by the first accepted candidate.

### Time to Resolution (TTR)

Mean repair time over successful repairs.

### Iteration Efficiency (IE)

Rewards successful repairs requiring fewer accepted repair attempts.

### Average Repair Iterations (ARI)

Mean number of accepted repair attempts for successful repairs.

### Failure Rate (FR)

Percentage of attempted cases that terminate without a verified repair.

Detailed definitions are provided in:

```text
docs/metrics.md
```

---

# Final Evaluation

The final experiment contains:

```text
8 models × 67 benchmarks = 536 model–benchmark evaluations
```

Overall model performance:

| Model | SR (%) | Pass@1 (%) |
| --- | ---: | ---: |
| Gemma 4 12B | 98.5 | 91.0 |
| Ornith 9B | 97.0 | 85.1 |
| Gemma 4 E4B | 95.5 | 88.1 |
| DeepSeek-Coder 6.7B | 83.6 | 64.2 |
| Llama 3.1 8B | 80.6 | 67.2 |
| Qwen2.5-Coder 7B | 77.6 | 71.6 |
| Qwen2.5-Coder 3B | 71.6 | 59.7 |
| Granite Code 8B | 52.2 | 46.3 |

Aggregate repair success decreased as fault interaction increased:

| Difficulty | SR (%) |
| --- | ---: |
| T1 Single-Fault | 91.2 |
| T2 Intermediate | 63.8 |
| T3 Advanced | 35.4 |

Across the 96 unsuccessful evaluations, repeated-candidate stagnation was the
dominant terminal failure mode.

The default model for standalone CLEAR repairs is therefore:

```text
gemma4:12b
```

This is a project-specific configuration decision based on the final CLEAR
experiment rather than a general claim that the model is universally optimal
for automated program repair.

---

# Failure Handling

CLEAR distinguishes repair failure from infrastructure failure.

Terminal model/repair outcomes include:

- repeated-candidate stagnation;
- model protocol failure;
- model generation-limit exhaustion;
- repair-budget exhaustion.

A normal failing pytest execution is not necessarily terminal. It can instead
produce feedback for the next candidate-generation attempt.

Infrastructure failures are recorded separately so that benchmark or execution
environment faults do not silently become model-performance failures.

See:

```text
docs/failure_taxonomy.md
```

---

# Outputs

Benchmark runs create timestamped experiment directories beneath:

```console
tests/logs/
```

For example:

```console
tests/logs/
└── run_<timestamp>_<models>_<tiers>_<categories>_automated_repair/
```

Generated outputs may include:

- raw CSV dataset;
- raw JSON dataset;
- aggregate CSV tables;
- LaTeX tables;
- Markdown analysis report;
- failure summaries;
- generated academic figures.

Full benchmark runs are primarily aggregate experimental evaluations.
Standalone `--save-run` executions are intended for retaining individual
repair artefacts such as exact patches and repaired source files.

---

# Documentation

The complete project documentation is stored in:

```text
docs/
```

Available documentation includes:

## Getting Started

1. [`index.md`](docs/index.md)
2. [`cli_reference.md`](docs/cli_reference.md)
3. [`models.md`](docs/models.md)
4. [`benchmark_spec.md`](docs/benchmark_spec.md)

## Evaluation and Results

- [`metrics.md`](docs/metrics.md)
- [`model_matrix.md`](docs/model_matrix.md)
- [`failure_taxonomy.md`](docs/failure_taxonomy.md)
- [`results_schema.md`](docs/results_schema.md)
- [`reproducibility.md`](docs/reproducibility.md)

## Development and Operations

- [`troubleshooting.md`](docs/troubleshooting.md)
- [`contributing.md`](docs/contributing.md)

Users of the packaged CLI can view the same documentation in the project
repository without needing to inspect the implementation source.

Command-specific documentation is also available directly through:

```console
clear-repair --help
clear-repair repair --help
clear-repair benchmark --help
clear-repair models --help
clear-repair doctor --help
```

---

# Project Structure

```console
CLEAR/
├── docs/
│   ├── index.md
│   ├── benchmark_spec.md
│   ├── cli_reference.md
│   ├── contributing.md
│   ├── failure_taxonomy.md
│   ├── metrics.md
│   ├── model_matrix.md
│   ├── models.md
│   ├── reproducibility.md
│   ├── results_schema.md
│   └── troubleshooting.md
│
├── src/
│   ├── agent/
│   ├── benchmarking/
│   ├── cli/
│   ├── core/
│   ├── repair/
│   ├── reporting/
│   ├── tools/
│   ├── utils/
│   └── main.py
│
├── tests/
│   ├── benchmarks/
│   │   ├── benchmark_manifest.json
│   │   ├── single_fault/
│   │   ├── intermediate/
│   │   └── advanced/
│   ├── logs/
│   └── validate_benchmarks.py
│
├── dist/
│   └── clear_repair-0.1.0-py3-none-any.whl
│
├── Dockerfile
├── pyproject.toml
├── run_benchmarks.py
├── run_benchmarks.bat
└── README.md
```

---

# Reproducibility

For full experimental reproduction, use the source repository rather than
only the packaged application.

Recommended procedure:

```console
git clone https://github.com/rrburman23/CLEAR.git
cd CLEAR

python -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install -e .

docker build -t clear-executor .

python tests/validate_benchmarks.py

clear-repair doctor
clear-repair models
```

Then run the required benchmark configuration:

```console
clear-repair benchmark `
    --models gemma4:12b `
    --difficulties single_fault `
    --types logic
```

The final dissertation experiment used one evaluation for each
model–benchmark pair under a common inference and verification protocol.

See:

```text
docs/reproducibility.md
```

---

# Development

Compile the Python source:

```console
python -m compileall src run_benchmarks.py
```

Validate benchmarks:

```console
python tests/validate_benchmarks.py
```

Install the development package:

```console
python -m pip install -e .
```

Build release distributions:

```console
python -m pip install build
python -m build
```

This creates:

```console
dist/
├── clear_repair-0.1.0-py3-none-any.whl
└── clear_repair-0.1.0.tar.gz
```

A release wheel should be tested inside a clean virtual environment before
distribution.

---

# Research Context

CLEAR was developed as an MSc Artificial Intelligence dissertation project at
Queen Mary University of London.

The project investigates whether locally deployable language models can
perform bounded, test-verified autonomous program repair under a shared
execution protocol, and how repair behaviour changes as seeded fault
complexity increases.

The final evaluation includes:

- 67 benchmark programs;
- 268 human-authored pytest tests;
- 105 seeded faults;
- three repair-difficulty tiers;
- eight principal local language models;
- 536 model–benchmark evaluations.

---

## Author

**Rohan Burman**  
MSc Artificial Intelligence  
Queen Mary University of London  
2026
