Metadata-Version: 2.4
Name: pyciaf
Version: 1.7.0
Summary: Cognitive Insight Audit Framework - A comprehensive framework for verifiable AI training and inference pipelines with advanced watermarking
Author-email: Denzil James Greenwood <founder@cognitiveinsight.ai>
Maintainer-email: Denzil James Greenwood <founder@cognitiveinsight.ai>
Project-URL: Homepage, https://github.com/DenzilGreenwood/pyciaf
Project-URL: Repository, https://github.com/DenzilGreenwood/pyciaf
Project-URL: Documentation, https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/index.md
Project-URL: Bug Tracker, https://github.com/DenzilGreenwood/pyciaf/issues
Project-URL: Changelog, https://github.com/DenzilGreenwood/pyciaf/blob/main/CHANGELOG.md
Keywords: ai,artificial intelligence,machine learning,cryptography,provenance,compliance,gdpr,ai-act,nist,hipaa,audit,transparency,explainable-ai,uncertainty-quantification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security :: Cryptography
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cryptography>=3.4.8
Requires-Dist: blake3>=0.3.3
Requires-Dist: numpy>=1.21.0
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: Pillow>=10.2.0
Requires-Dist: ImageHash>=4.3.2
Requires-Dist: opencv-python>=4.8.0
Requires-Dist: boto3>=1.26.0
Requires-Dist: azure-storage-blob>=12.14.0
Provides-Extra: ml
Requires-Dist: scikit-learn>=1.0.0; extra == "ml"
Requires-Dist: pandas>=1.3.0; extra == "ml"
Provides-Extra: explainability
Requires-Dist: shap>=0.41.0; extra == "explainability"
Requires-Dist: lime>=0.2.0.1; extra == "explainability"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.5.0; extra == "viz"
Requires-Dist: seaborn>=0.11.0; extra == "viz"
Requires-Dist: plotly>=5.0.0; extra == "viz"
Provides-Extra: audio-advanced
Requires-Dist: librosa>=0.10.0; extra == "audio-advanced"
Requires-Dist: soundfile>=0.12.0; extra == "audio-advanced"
Provides-Extra: multiformat
Requires-Dist: ffmpeg-python>=0.2.0; extra == "multiformat"
Provides-Extra: audio-fingerprint
Requires-Dist: pyacoustid>=2.3.0; extra == "audio-fingerprint"
Provides-Extra: web
Requires-Dist: flask>=2.0.0; extra == "web"
Requires-Dist: flask-cors>=3.0.10; extra == "web"
Provides-Extra: pdf
Requires-Dist: reportlab>=3.6.0; extra == "pdf"
Provides-Extra: full
Requires-Dist: scikit-learn>=1.0.0; extra == "full"
Requires-Dist: shap>=0.41.0; extra == "full"
Requires-Dist: lime>=0.2.0.1; extra == "full"
Requires-Dist: matplotlib>=3.5.0; extra == "full"
Requires-Dist: seaborn>=0.11.0; extra == "full"
Requires-Dist: pandas>=1.3.0; extra == "full"
Requires-Dist: plotly>=5.0.0; extra == "full"
Requires-Dist: reportlab>=3.6.0; extra == "full"
Provides-Extra: watermarking-full
Requires-Dist: librosa>=0.10.0; extra == "watermarking-full"
Requires-Dist: soundfile>=0.12.0; extra == "watermarking-full"
Requires-Dist: ffmpeg-python>=0.2.0; extra == "watermarking-full"
Requires-Dist: pyacoustid>=2.3.0; extra == "watermarking-full"
Provides-Extra: dev
Requires-Dist: pytest>=6.0.0; extra == "dev"
Requires-Dist: pytest-cov>=2.12.0; extra == "dev"
Requires-Dist: black>=21.0.0; extra == "dev"
Requires-Dist: flake8>=3.9.0; extra == "dev"
Requires-Dist: mypy>=0.910; extra == "dev"
Requires-Dist: pre-commit>=2.15.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=4.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0.0; extra == "docs"
Requires-Dist: myst-parser>=0.15.0; extra == "docs"
Provides-Extra: all
Requires-Dist: scikit-learn>=1.0.0; extra == "all"
Requires-Dist: shap>=0.41.0; extra == "all"
Requires-Dist: lime>=0.2.0.1; extra == "all"
Requires-Dist: matplotlib>=3.5.0; extra == "all"
Requires-Dist: seaborn>=0.11.0; extra == "all"
Requires-Dist: pandas>=1.3.0; extra == "all"
Requires-Dist: plotly>=5.0.0; extra == "all"
Requires-Dist: reportlab>=3.6.0; extra == "all"
Requires-Dist: librosa>=0.10.0; extra == "all"
Requires-Dist: soundfile>=0.12.0; extra == "all"
Requires-Dist: ffmpeg-python>=0.2.0; extra == "all"
Requires-Dist: pyacoustid>=2.3.0; extra == "all"
Requires-Dist: flask>=2.0.0; extra == "all"
Requires-Dist: flask-cors>=3.0.10; extra == "all"
Dynamic: license-file

# CIAF – Cognitive Insight Audit Framework

**Version:** 1.4.0 (Development)

> **✅ PRODUCTION-CAPABLE RELEASE**
> CIAF v1.3.1 introduces **Agentic Execution Boundaries** - zero-trust IAM/PAM controls for autonomous AI agents. This release extends CIAF's cryptographic provenance with identity management, privilege elevation, and tamper-evident audit trails specifically designed for agent systems. All previous features remain production-capable with comprehensive enterprise features.
>
> **Current Version:** v1.4.0 is in active development. Latest stable release: v1.3.1

A Python framework for verifiable AI training and inference with cryptographic provenance, selective ("lazy") capsule materialization, and compliance mapping.

![License: BUSL-1.1](https://img.shields.io/badge/License-BUSL--1.1-blue.svg)
![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)
![Code Style: Black](https://img.shields.io/badge/code%20style-Black-000000.svg)
[![Security Policy](https://img.shields.io/badge/Security-Policy-informational.svg)](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/SECURITY.md)

---

## Overview

CIAF (Cognitive Insight Audit Framework) addresses AI transparency, auditability, and compliance in production. It provides cryptographically verifiable provenance tracking, **Lazy Capsule Materialization (LCM)**, and audit artifacts designed to map to major regulatory frameworks.

### Key Features

- **Cryptographic Provenance Tracking** — End-to-end verifiable data lineage with Merkle trees and hash connections.
- **Lazy Capsule Materialization (LCM)** — On-demand proof capsule materialization to minimize storage and exposure.
- **Agentic Execution Boundaries** (NEW in v1.3.0) — Zero-trust IAM/PAM controls for autonomous AI agents with cryptographic audit trails.
- **AI Forensic Provenance Layer** (v1.2.0) — Dual-state watermarking with hierarchical verification, forensic fragment detection, and tamper evidence. Production-capable for text, beta for images.
- **Web AI Governance** (v1.2.0) — Browser-based AI usage detection, policy enforcement, and shadow AI monitoring with cryptographic receipts.
- **Compliance Mapping** — Artifacts designed to map to EU AI Act, NIST AI RMF, GDPR/HIPAA, SOX, ISO/IEC 27001 (see `docs/compliance/`).
- **Security-First Design** — Optional AES-256-GCM, secure anchor derivation, tamper-evident audit trails.
- **Risk Assessment Patterns** — Bias/fairness checks and uncertainty-quantification scaffolding.
- **Transparency & Explainability** — Hooks for decision transparency and receipt generation.
- **Healthcare Patterns** — PHI minimization and consent-tracking patterns (final compliance depends on deployment).
- **Performance Monitoring** — Basic metrics for LCM operations.
- **Metadata Traceability** — Complete inference-to-model lineage tracking with single receipt lookup.

---

## Installation

### Option A: From source
```bash
git clone https://github.com/DenzilGreenwood/pyciaf.git
cd pyciaf
python -m venv .venv
# Windows: .venv\Scripts\activate
source .venv/bin/activate
pip install -U pip build
pip install -e .
```

### Option B: Built Package (Recommended)
```bash
# Install from built wheel (fastest)
pip install ciaf-1.1.0-py3-none-any.whl

# Or install from source distribution
pip install ciaf-1.1.0.tar.gz
```

### Option C: Directly from GitHub
```bash
pip install "git+https://github.com/DenzilGreenwood/pyciaf.git#egg=ciaf"
```

---

## 📚 Documentation

CIAF v1.3.1 includes comprehensive documentation consolidated in the `docs/` directory:

| Document | Description |
|----------|-------------|
| **[docs/index.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/index.md)** | Main documentation hub with navigation |
| **[docs/quickstart.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/quickstart.md)** | 5-minute getting started guide |
| **[docs/concepts.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/concepts.md)** | Core architectural concepts and anchor system |
| **[docs/receipts.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/receipts.md)** | Receipt formats and verification process |
| **[docs/compliance-mapping.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/compliance-mapping.md)** | Regulatory framework mappings (EU AI Act, NIST, GDPR, etc.) |
| **[docs/DEFERRED_LCM_README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/DEFERRED_LCM_README.md)** | High-performance LCM implementation guide |
| **[docs/DEPLOYABLE_MODEL_DEMO_GUIDE.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/DEPLOYABLE_MODEL_DEMO_GUIDE.md)** | Production deployment patterns |
| **[docs/MODEL_BUILDING_GUIDE_V1_1_0.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/MODEL_BUILDING_GUIDE_V1_1_0.md)** | Complete model building guide |
| **[docs/CODING_STANDARDS.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/CODING_STANDARDS.md)** | Development standards and guidelines |
| **[docs/WHITEPAPER.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/WHITEPAPER.md)** | Technical whitepaper and research foundations |

### Module-Specific Documentation

| Module | Documentation | Description |
|--------|---------------|-------------|
| **Agentic Execution Boundaries** | [ciaf/agents/README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/agents/README.md) | Zero-trust IAM/PAM for autonomous agents |
| | [docs/agents/QUICKSTART.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/agents/QUICKSTART.md) | 5-minute agents tutorial |
| | [docs/agents/DEVELOPER_GUIDE.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/agents/DEVELOPER_GUIDE.md) | Complete integration guide |
| **AI Forensic Provenance** | [ciaf/watermarks/README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/watermarks/README.md) | Dual-state watermarking & forensic verification |
| | [docs/HIERARCHICAL_VERIFICATION_STRATEGY.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/HIERARCHICAL_VERIFICATION_STRATEGY.md) | Three-tier verification architecture |
| | [docs/HIERARCHICAL_VERIFICATION_QUICK_REFERENCE.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/HIERARCHICAL_VERIFICATION_QUICK_REFERENCE.md) | 30-second watermark quick reference |
| | [WATERMARK_TECHNICAL_ASSESSMENT.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/WATERMARK_TECHNICAL_ASSESSMENT.md) | Technical review & known issues |
| **Web AI Governance** | [ciaf/web/README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/web/README.md) | Browser-based AI usage governance |

### Quick Navigation

- **New to CIAF?** → Start with [docs/quickstart.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/quickstart.md)
- **Agentic systems?** → See [docs/agents/QUICKSTART.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/agents/QUICKSTART.md) and [examples/agents_scenarios/](https://github.com/DenzilGreenwood/pyciaf/tree/main/examples/agents_scenarios)
- **AI forensic provenance?** → Check [ciaf/watermarks/README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/watermarks/README.md) (text production-capable, see known issues)
- **Web AI governance?** → Read [ciaf/web/README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/web/README.md)
- **Production deployment?** → See [docs/MODEL_BUILDING_GUIDE_V1_1_0.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/MODEL_BUILDING_GUIDE_V1_1_0.md)
- **Performance optimization?** → Check [docs/DEFERRED_LCM_README.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/DEFERRED_LCM_README.md)
- **Compliance requirements?** → Review [docs/compliance-mapping.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/compliance-mapping.md)
- **Technical architecture?** → Read [docs/concepts.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/concepts.md) and [docs/WHITEPAPER.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/WHITEPAPER.md)

## Project Structure

CIAF v1.3.1 follows a clean, professional project structure:

```
PYPI/                           # Root project directory
├── ciaf/                       # Main CIAF package
│   ├── core/                   # Core functionality
│   ├── api/                    # High-level API
│   ├── lcm/                    # Lifecycle Management
│   ├── compliance/             # Regulatory compliance
│   ├── wrappers/               # Model wrappers
│   ├── agents/                 # Agentic execution boundaries (v1.3.0)
│   ├── watermarks/             # AI watermarking & verification (v1.2.0)
│   ├── web/                    # Web AI governance (v1.2.0)
│   └── ...                     # Additional modules
├── examples/                   # Usage examples and demos
│   ├── agents_scenarios/       # Healthcare, financial, production scenarios
│   ├── hierarchical_verification_examples.py  # Watermark verification
│   └── ...                     # Additional examples
├── tests/                      # Comprehensive test suite
├── docs/                       # Complete documentation
│   ├── agents/                 # Agentic execution documentation
│   ├── HIERARCHICAL_VERIFICATION_*.md  # Watermarking docs
│   └── ...                     # Core documentation
├── tools/                      # Development utilities
└── PROJECT_STRUCTURE.md        # Detailed structure guide
```

See [PROJECT_STRUCTURE.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/docs/PROJECT_STRUCTURE.md) for complete details.

### Option C: PyPI (when published)
```bash
pip install pyciaf
```

---

## Quick Start

```python
from ciaf import CIAFFramework, ModelMetadataManager

framework = CIAFFramework("MyAI_Project")

# 1) Create a dataset anchor (cryptographic root for dataset operations)
anchor = framework.create_dataset_anchor(
    dataset_id="healthcare_data",
    dataset_metadata={"source": "hospital_system", "type": "medical_records"},
    master_password="secure_password_123"
)

# 2) Create provenance capsules for your data
data_items = [
    {"content": "patient_record_1", "metadata": {"id": "p001", "consent": True}},
    {"content": "patient_record_2", "metadata": {"id": "p002", "consent": True}},
]
capsules = framework.create_provenance_capsules("healthcare_data", data_items)

# 3) Create a model anchor (immutable parameter/architecture fingerprints + dataset authorization)
model_anchor = framework.create_model_anchor(
    model_name="diagnostic_model",
    model_parameters={"epochs": 100, "lr": 0.001},
    model_architecture={"type": "bert_classifier", "hidden": 768},
    authorized_datasets=["healthcare_data"],
    master_password="secure_model_password"
)

# 4) Produce a verifiable training snapshot
snapshot = framework.train_model(
    model_name="diagnostic_model",
    capsules=capsules,
    maa=model_anchor,
    training_params={"epochs": 100, "lr": 0.001},
    model_version="v1.0"
)

# 5) Validate integrity
assert framework.validate_training_integrity(snapshot)
print("Training integrity verified.")
```

---

## Architecture

```scss
CIAF Framework
├─ Core Components
│  ├─ Cryptographic Utilities (AES-256-GCM, SHA-256, HMAC)
│  ├─ Anchor Management (hierarchical anchor derivation)
│  └─ Merkle Tree Implementation
├─ Anchoring System
│  ├─ Dataset Anchors (Master → Dataset → Capsule)
│  └─ Lazy Managers (selective materialization)
├─ Provenance Tracking
│  ├─ Provenance Capsules (content + metadata)
│  └─ Training Snapshots (verifiable model states)
├─ Compliance Engine
│  ├─ Regulatory Mapping (EU AI Act, NIST, GDPR/HIPAA, etc.)
│  ├─ Validators (automated checks, where implemented)
│  └─ Audit Trails (append-only/WORM)
├─ Risk Assessment
│  ├─ Bias & Fairness patterns
│  ├─ Uncertainty-quantification scaffolding
│  └─ Security-assessment hooks
├─ Inference Management
│  ├─ Inference Receipts (verifiable prediction records)
│  ├─ ZKE Connections (privacy-preserving audit connections)
│  └─ Metadata Reveal (complete lineage tracing)
├─ Agentic Execution Boundaries (v1.3.0)
│  ├─ IAM Store (identity & role management)
│  ├─ PAM Store (JIT privilege elevation)
│  ├─ Policy Engine (RBAC + ABAC evaluation)
│  ├─ Evidence Vault (cryptographic receipts)
│  └─ Tool Executor (mediated invocation)
├─ AI Watermarking & Verification (v1.2.0)
│  ├─ Hierarchical Verification (3-tier strategy)
│  ├─ Forensic Fragments (DNA sampling)
│  ├─ Text/Image/Video Watermarking
│  ├─ Tamper Detection (hash chains)
│  └─ Perceptual Similarity (SimHash)
├─ Web AI Governance (v1.2.0)
│  ├─ AI Tool Detection (15+ tools, shadow AI)
│  ├─ Content Classification (PII, PHI, secrets)
│  ├─ Policy Engine (allow/warn/block/redact)
│  ├─ Event Collection (browser integration)
│  ├─ Cryptographic Receipts (evidence)
│  └─ Vault Storage (searchable governance)
├─ Metadata Management
│  ├─ Storage backends (JSON, SQLite, Pickle)
│  ├─ Configuration templates
│  └─ Integration utilities
└─ Utilities
   ├─ CLI Tools
   ├─ Model Wrappers
   └─ ML Framework Simulators
```

---

## Compliance Support

**Compliance Mapping:** CIAF's audit artifacts are designed to map to control intents across EU AI Act, NIST AI RMF, GDPR/HIPAA, SOX, ISO/IEC 27001. Coverage varies by control and typically requires organizational process overlays. See `docs/compliance/` for current status and gaps. **This is not legal advice.**

---

## Advanced Features

### Lazy Capsule Materialization (LCM)

Materialize only what you need, when you need it—while preserving cryptographic verifiability.

```python
# Create dataset anchor with a lazy manager
anchor = framework.create_dataset_anchor(
    dataset_id="large_dataset",
    dataset_metadata={"size": "1TB", "type": "image_data"},
    master_password="secure_anchor_password"
)

# Access the dataset's lazy manager
lazy_manager = framework.lazy_managers["large_dataset"]

# Materialize a capsule on demand
capsule = lazy_manager.materialize_capsule("item_001")
```

### Enhanced Model Anchor System

Immutable parameter/architecture fingerprints and dataset authorization.

```python
model_anchor = framework.create_model_anchor(
    model_name="sentiment_classifier",
    model_parameters={"learning_rate": 2e-5, "batch_size": 16, "num_epochs": 3, "model_type": "bert_classifier"},
    model_architecture={"base_model": "bert-base-uncased", "num_labels": 3, "hidden_size": 768},
    authorized_datasets=["training_data_v1", "validation_data_v1"],
    master_password="secure_model_password"
)
print("Model fingerprint:", model_anchor["parameters_fingerprint"])
print("Architecture fingerprint:", model_anchor["architecture_fingerprint"])
```

### Complete Audit Flow Integration

```python
# 1) Train with complete audit
training_snapshot = framework.train_model_with_audit(
    model_name="sentiment_classifier",
    capsules=training_capsules,
    training_params=training_params,
    model_version="1.0.0",
    user_id="data_scientist_alice"
)

# 2) Perform inference with audit connections
receipt = framework.perform_inference_with_audit(
    model_name="sentiment_classifier",
    query="This product is amazing!",
    ai_output="positive (confidence: 0.95)",
    training_snapshot=training_snapshot,
    user_id="api_user"
)

# 3) Retrieve complete audit trail
audit_trail = framework.get_complete_audit_trail("sentiment_classifier")
print("Datasets:", audit_trail["verification"]["total_datasets"])
print("Audit records:", audit_trail["verification"]["total_audit_records"])
print("Inference receipts:", audit_trail["inference_connections"]["total_receipts"])
```

---

## Tools & Verification

CIAF includes a comprehensive suite of tools for demonstration, verification, and audit compliance located in the `tools/` directory.

### 🔧 Verification Tools

#### Independent Receipt Verification
```bash
# Verify any CIAF receipt with detailed cryptographic validation
cd tools/
python verify_receipt.py path/to/receipt.json
```

The verification tool provides detailed output including:
- **Dataset Merkle root validation** with expected vs calculated hashes
- **Model parameter fingerprints** with complete parameter display
- **Model architecture verification** with full architecture specs
- **Audit connection integrity** with hash chain validation for each event

#### Enhanced Verification Features
- ✅ **Complete hash transparency** - Shows expected vs calculated values for all cryptographic operations
- ✅ **Parameter visibility** - Displays full model configuration and architecture
- ✅ **Audit chain details** - Individual event validation with hash linking verification
- ✅ **Error diagnostics** - Clear indication of validation failures with specific details
- ✅ **Compliance ready** - Output suitable for regulatory audits and forensic investigation

### 🚀 Demo & Benchmarking Tools

#### Deferred LCM Performance Demo
```bash
cd tools/
python deferred_lcm_benchmark.py
```

This benchmark demonstrates:
- **Performance comparison** between standard CIAF, high-performance deferred LCM, and adaptive LCM
- **Real-world fraud detection** scenario with 1000+ predictions
- **Adaptive mode switching** based on system load and processing requirements
- **Comprehensive metrics** including throughput (samples/sec) and latency analysis

#### Receipt Verification Workflow Demo
```bash
cd tools/
python demo_receipt_verification.py
```

Complete workflow demonstration:
1. **Extracts receipts** from deferred LCM audit batches
2. **Converts to verifiable format** compatible with independent verification
3. **Runs verification** using the enhanced verification tool
4. **Shows detailed results** with full audit trail information

#### Receipt Extraction Tool
```bash
cd tools/
python extract_receipt_for_verification.py
```

Converts deferred LCM audit batches into standalone CIAF receipts for independent verification:
- **Merkle tree construction** from training data samples
- **Model fingerprint generation** for parameters and architecture
- **Audit chain creation** with proper hash linking
- **Deferred LCM metadata** preservation for compliance tracking

### 📊 Demo Features

The tools demonstrate:

#### **Enhanced Model Wrapper** (`enhanced_model_wrapper.py`)
- **Deferred LCM integration** with background audit materialization
- **Adaptive mode switching** between immediate and deferred processing
- **Performance optimization** while maintaining full compliance
- **Receipt generation** with lightweight audit creation

#### **Performance Benchmarking**
Example output from deferred LCM benchmark:
```
Performance Comparison Results:
=====================================
Standard CIAF: 0.0006s avg (1723 samples/sec)
High-performance: 0.0016s avg (625 samples/sec)  
Adaptive LCM: 0.0029s avg (548 samples/sec)

Audit Trail Generation: 50 receipts created
Verification: All receipts independently verified ✅
```

#### **Verification Transparency**
Example verification output:
```
🔍 Verifying CIAF Receipt...
========================================
📊 Dataset Merkle root: ✅ Valid
   📋 Dataset ID: deferred_lcm_demo_dataset
   🌿 Leaf count: 4
   🔍 Expected root: 3d1081642ad6c5e2f327f8f288dafaba...
   🧮 Calculated root: 3d1081642ad6c5e2f327f8f288dafaba...
🤖 Model parameters: ✅ Valid
   📝 Model name: Enhanced_CIAF_Demo_Model
   🔧 Parameters: {'model_type': 'RandomForestClassifier'...}
   🔍 Expected fingerprint: 94c603d9c0c024cf124ecd9dc136107b...
   🧮 Calculated fingerprint: 94c603d9c0c024cf124ecd9dc136107b...
📋 Audit connections: ✅ Valid
   🔗 Event count: 2
   📄 Event 1: training_started (✅)
      🆔 Event ID: training_start
      ⏰ Timestamp: 2025-09-19T10:00:00Z
      🔍 Expected hash: 3ae6ac3adf1cf3579d9c99fb4c1d52bf...
      🧮 Calculated hash: 3ae6ac3adf1cf3579d9c99fb4c1d52bf...
========================================
🎯 Overall Receipt: ✅ VALID
```

### 🎯 Usage Instructions

1. **Run the benchmark** to see deferred LCM performance improvements:
   ```bash
   cd tools/
   python deferred_lcm_benchmark.py
   ```

2. **Verify generated receipts** using the independent verification tool:
   ```bash
   python verify_receipt.py ../extracted_ciaf_receipt_for_verification.json
   ```

3. **Complete workflow demo** from generation to verification:
   ```bash
   python demo_receipt_verification.py
   ```

4. **Extract custom receipts** from any audit batch:
   ```bash
   python extract_receipt_for_verification.py
   ```

### 📁 Tools Directory Structure
```
tools/
├── verify_receipt.py              # Independent receipt verification
├── deferred_lcm_benchmark.py      # Performance demonstration  
├── enhanced_model_wrapper.py      # Enhanced CIAF wrapper
├── demo_receipt_verification.py   # Complete workflow demo
├── extract_receipt_for_verification.py  # Receipt extraction
├── verification_enhancement_summary.py  # Feature summary
└── examples/                      # Additional examples
    ├── quickstart.py
    ├── lcm_integration_demo.py
    └── credit_model_demo.py
```

These tools provide everything needed to:
- **Understand CIAF capabilities** through working demonstrations
- **Verify audit integrity** with independent cryptographic validation
- **Benchmark performance** across different LCM configurations
- **Generate compliance reports** suitable for regulatory review
- **Debug verification issues** with detailed diagnostic output

---

## CLI Tools

```bash
# Setup metadata storage
python -m ciaf.cli setup my_project --backend sqlite --template production

# Generate a compliance report
python -m ciaf.cli compliance eu_ai_act my_model_id --format html --output compliance_report.html

# Trace metadata lineage from inference receipt
python -m ciaf.examples.metadata_reveal
```

---

## Integration Examples

### Scikit-learn

```python
from ciaf import CIAFModelWrapper
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier()
wrapped = CIAFModelWrapper(model, "fraud_detection_v1")
wrapped.fit(X_train, y_train)
preds = wrapped.predict(X_test)
```

### Metadata Lineage Tracing

```python
from ciaf.examples.metadata_reveal import MetadataReveal

# Trace complete lineage from single inference receipt
revealer = MetadataReveal()
trail = revealer.reveal_metadata_trail("r_a1b2c3d4")

# Verify integrity and generate compliance report
integrity_ok = revealer.verify_trail_integrity(trail)
report = revealer.export_trail_report(trail, "compliance_report.json")
```

### TensorFlow / PyTorch (simulated)

```python
from ciaf.simulation import MLFrameworkSimulator

sim = MLFrameworkSimulator("neural_network")
training_snapshot = sim.train_model(
    training_data_capsules=capsules,
    maa=model_anchor,
    training_params={"epochs": 50, "batch_size": 32},
    model_version="v2.0"
)
```

---

## Performance & Metrics

```python
metrics = framework.get_performance_metrics("my_dataset")
print("Materialization rate:", f"{metrics['materialization_rate']:.2%}")
print("Total items:", metrics["total_items"])
print("Materialized capsules:", metrics["materialized_capsules"])
```

---

## Security

> **See our [Security Policy](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/SECURITY.md) for reporting vulnerabilities, supported versions, and secure deployment guidance.**

### Cryptographic Security

- **AES-256-GCM** (optional) for authenticated encryption (supports AAD).
- **SHA-256** for integrity hashing.
- **HMAC-SHA-256** for anchor derivation and message authentication.
- **Merkle Trees** with canonical concatenation for tamper-evident sets.

### Anchor Management

- Hierarchical anchor derivation (Master → Dataset → Capsule).
- Cryptographically secure randomness and high-entropy binary anchors.
- Canonicalized operations for derivations & Merkle policies.
- Backwards compatibility for legacy key-based terminology.

### Access Controls (patterns)

- Role-based access patterns.
- Audit-logging hooks.
- Session-management scaffolding.

---

## Healthcare & HIPAA Patterns

```python
from ciaf import ModelMetadataManager
from ciaf.compliance import ComplianceFramework

manager = ModelMetadataManager("healthcare_ai", "1.0.0")
manager.enable_phi_protection()
manager.set_compliance_frameworks([ComplianceFramework.HIPAA])
manager.capture_metadata({
    "patient_id": "XXXXX",  # handled with PHI patterns
    "diagnosis": "diabetes",
    "consent_status": "active"
})
```

> **Note:** CIAF provides patterns for PHI minimization and consent tracking. Final compliance depends on your deployment architecture, governance, and policies.

---

## Contributing

We welcome contributions!

1. **Code Style** — Black
2. **Testing** — Add tests; ensure all pass
3. **Docs** — Update documentation for any API changes
4. **Security** — Follow secure coding practices; report issues via `SECURITY.md`

### Development Setup

```bash
git clone https://github.com/DenzilGreenwood/pyciaf.git
cd pyciaf
pip install -e .
```

---

## Support & Community

- **Documentation**: [https://ciaf.readthedocs.io](https://ciaf.readthedocs.io)
- **Issues**: [https://github.com/DenzilGreenwood/pyciaf/issues](https://github.com/DenzilGreenwood/pyciaf/issues)
- **Discussions**: [https://github.com/DenzilGreenwood/pyciaf/discussions](https://github.com/DenzilGreenwood/pyciaf/discussions)
- **Security**: See [ciaf/SECURITY.md](https://github.com/DenzilGreenwood/pyciaf/blob/main/ciaf/SECURITY.md)

---

## Status & Roadmap

### Current Status

| Feature | Status | Notes |
|---------|--------|-------|
| **Core Framework** | ✅ Working | Anchoring + LCM |
| **Cryptographic Primitives** | ✅ Working | SHA-256, HMAC, AES-GCM |
| **Merkle Trees** | ✅ Working | Deterministic proofs |
| **Dataset Anchoring** | ✅ Working | Hierarchical derivation |
| **Model Anchoring** | ✅ Working | Param/arch fingerprints |
| **Audit Trails** | ✅ Working | Hash-connected events |
| **Lazy Materialization** | ✅ Working | On-demand capsules |
| **Inference Connections** | ✅ Working | ZKE connections system |
| **Metadata Traceability** | ✅ Working | Complete lineage tracking |
| **Basic CLI** | 🧪 Prototype | Setup & compliance |
| **Compliance Mapping** | 🧪 Prototype | EU AI Act, NIST |
| **Receipt Verification** | ✅ Working | Independent verifier |
| **Healthcare Patterns** | 🧪 Prototype | PHI scaffolding |

### Near-term Roadmap 

| Feature | Priority | Target |
|---------|----------|--------|
| **API Stabilization** | 🔴 High | Finalize public APIs |
| **Documentation** | 🔴 High | Complete API reference |
| **Test Coverage** | 🔴 High | >90% |
| **Performance Optimization** | 🟡 Medium | LCM efficiency |
| **CLI Enhancement** | 🟡 Medium | Full-featured CLI |

### Medium-term Roadmap 

| Feature | Priority | Target |
|---------|----------|--------|
| **GDPR/HIPAA Compliance** | 🔴 High | Production-ready patterns |
| **Advanced Analytics** | 🟡 Medium | Bias/fairness metrics |
| **Integration Libraries** | 🟡 Medium | TF/PyTorch wrappers |
| **Web Dashboard** | 🟢 Low | Audit visualization |
| **Enterprise Features** | 📋 Planned | SSO, RBAC, deployment |

### Research Areas

- **Zero-Knowledge Proofs** (ZK-SNARKs) for privacy-preserving verification
- **Immutable Audit Ledgers** for tamper-evident audit storage 
- **Homomorphic encryption** for computation on encrypted data
- **Formal verification** of cryptographic correctness

---

## License

This project is licensed under the **Business Source License 1.1 (BUSL-1.1)** — see [LICENSE](LICENSE) for full details.

**Key Terms:**
- ✅ **Non-commercial use**: Academic research, evaluation, personal use, and open source contributions are permitted
- ✅ **90-day evaluation**: Internal business evaluation and testing allowed
- ✅ **Change Date**: January 1, 2029 - automatically converts to Apache License 2.0
- ⚠️ **Commercial use**: Requires a commercial license from CognitiveInsight.ai
- ⚠️ **Production deployments**: Not permitted without a commercial license (beyond 90-day evaluation)
- ⚠️ **SaaS/Hosted services**: Requires commercial license

**Trademarks:** "Cognitive Insight™" and "LCM™" (Lazy Capsule Materialization) are trademarks of Denzil James Greenwood.

For commercial licensing inquiries: 📧 founder@cognitiveinsight.ai
Website: https://cognitiveinsight.ai

---

## Acknowledgments

- **cryptography** library and the broader Python security ecosystem
- **Regulatory frameworks**: EU AI Act, NIST AI RMF, GDPR/HIPAA, ISO/IEC 27001, SOX (for mapping inspiration)

> **Personal note:** This project is a work in progress and reflects a commitment to secure, verifiable, and compliant AI systems. The framework is updated periodically as needed to maintain relevance with evolving regulatory requirements and technological advances. Feedback is highly appreciated!
> 
> *— Denzil James Greenwood*
