# OmniClaude Scripts Directory

Comprehensive system verification and testing scripts for OmniClaude infrastructure.

## Quick Reference

### System Verification Scripts

| Script | Purpose | Runtime | Exit Code |
|--------|---------|---------|-----------|
| `verify_database_system.sh` | **Comprehensive database verification** (70 checks) | ~5-10s | 0=pass, 1=fail, 2=fatal |
| `health_check.sh` | Service health and infrastructure status | ~3-5s | 0=healthy, 1=issues |
| `test_system_functionality.sh` | **Master test suite** (runs all tests) | ~15-30s | 0=pass, 1=fail, 2=fatal |
| `validate-env.sh` | Environment configuration validation | ~1s | 0=valid, 1=invalid |

### Individual Component Tests (tests/ directory)

| Script | Purpose | What It Tests |
|--------|---------|---------------|
| `test_postgres_functionality.sh` | PostgreSQL basic functionality | Connectivity, CRUD operations, performance |
| `test_kafka_functionality.sh` | Kafka/Redpanda message bus | Topics, producers, consumers, messages |
| `test_intelligence_functionality.sh` | Intelligence integration | Archon services, Qdrant, pattern discovery |
| `test_routing_functionality.sh` | Agent routing service | Event-based routing, recommendations |

## Common Workflows

### 1. Pre-Commit Check (Fast)

```bash
# Run environment validation only (~1 second)
./scripts/validate-env.sh .env
```

### 2. Development Check (Medium)

```bash
# Run comprehensive database verification (~5-10 seconds)
./scripts/verify_database_system.sh

# View detailed report
cat /tmp/database_verification_latest.txt
```

### 3. Full System Verification (Comprehensive)

```bash
# Run ALL tests (~15-30 seconds)
./scripts/test_system_functionality.sh

# Includes:
#   - Kafka message bus test
#   - PostgreSQL database test
#   - Database system verification (70 checks)
#   - Intelligence integration test
#   - Agent routing test
```

### 4. Health Check (Service Status)

```bash
# Check service health and recent activity (~3-5 seconds)
./scripts/health_check.sh

# Output files:
#   - /tmp/health_check_latest.txt (latest)
#   - /tmp/health_check_history.log (history)
```

## Detailed Script Documentation

### verify_database_system.sh

**NEW**: Comprehensive database system verification with 70 checks across 12 categories.

**What it checks**:
- ✅ All 30 tables exist and have correct schema
- ✅ All 9 critical views function correctly
- ✅ CRUD operations (Create, Read, Update, Delete)
- ✅ Data integrity (foreign keys, indexes, constraints)
- ✅ All 4 action types (tool_call, decision, error, success)
- ✅ Recent activity (last 24 hours)
- ✅ Performance (query latency, connection pool)
- ✅ UUID handling (type validation, insertion, retrieval)
- ✅ Edge cases (NULL constraints, timestamps, large text)

**Usage**:
```bash
./scripts/verify_database_system.sh

# View report
cat /tmp/database_verification_latest.txt
```

**Documentation**: See `DATABASE_VERIFICATION_GUIDE.md` for complete details.

**Health Score**:
- ≥95%: Excellent
- 85-94%: Good
- 70-84%: Acceptable
- <70%: Poor

### health_check.sh

System-wide health check for all OmniClaude services.

**What it checks**:
- Docker service status (archon-*, omninode-*)
- Kafka connectivity and topics
- Qdrant collections and vector counts
- PostgreSQL connectivity and table counts
- Recent manifest injection quality
- Intelligence collection status

**Usage**:
```bash
./scripts/health_check.sh

# View latest
cat /tmp/health_check_latest.txt

# View history
tail -100 /tmp/health_check_history.log
```

### test_system_functionality.sh

Master test suite that orchestrates all functional tests.

**What it runs**:
1. Kafka Message Bus test
2. PostgreSQL Database test
3. **Database System Verification** (70 checks)
4. Intelligence Integration test
5. Agent Routing test

**Usage**:
```bash
./scripts/test_system_functionality.sh

# Summary output:
#   Total Tests: 5
#   Passed: 5
#   Failed: 0
#   System Health: 100%
```

### validate-env.sh

Validates environment configuration for safety and completeness.

**What it checks**:
- Required variables are set
- No placeholder values remain
- Password strength requirements
- URL format validation
- Port number ranges (1-65535)

**Usage**:
```bash
./scripts/validate-env.sh .env

# Or specific environment
./scripts/validate-env.sh .env.prod
```

## Exit Codes

All scripts follow consistent exit code conventions:

| Exit Code | Meaning | Action |
|-----------|---------|--------|
| 0 | Success | All checks passed (warnings OK) |
| 1 | Failure | One or more checks failed |
| 2 | Fatal Error | Cannot proceed (missing .env, no connectivity) |

## Output Files

Scripts generate reports in `/tmp/`:

| File | Generated By | Contains |
|------|--------------|----------|
| `/tmp/database_verification_latest.txt` | `verify_database_system.sh` | Health score and summary |
| `/tmp/health_check_latest.txt` | `health_check.sh` | Latest health status |
| `/tmp/health_check_history.log` | `health_check.sh` | Appended history |

## Integration with CI/CD

### GitHub Actions Example

```yaml
- name: Validate Environment
  run: ./scripts/validate-env.sh .env

- name: Run System Tests
  run: ./scripts/test_system_functionality.sh
```

### Pre-Commit Hook Example

```bash
#!/bin/bash
# .git/hooks/pre-commit

echo "Running database verification..."
./scripts/verify_database_system.sh

if [ $? -ne 0 ]; then
    echo "❌ Database verification failed. Commit aborted."
    exit 1
fi

echo "✅ Database verification passed"
```

## Troubleshooting

### "Cannot connect to database"

```bash
# 1. Verify .env file exists
ls -la .env

# 2. Source environment
source .env

# 3. Test manual connection
psql -h 192.168.86.200 -p 5436 -U postgres -d omnibase_infra
```

### "Missing tables" errors

```bash
# 1. Check database name
source .env && echo $POSTGRES_DATABASE

# 2. List actual tables
psql -h 192.168.86.200 -p 5436 -U postgres -d omnibase_infra -c "\dt"

# 3. Run schema migrations if needed
# (see database migration documentation)
```

### "Service not responding" errors

```bash
# 1. Check Docker services
docker ps | grep -E "(archon|omninode)"

# 2. Restart specific service
docker restart archon-intelligence

# 3. Check service logs
docker logs -f archon-intelligence
```

## Adding New Tests

### 1. Create Test Script

```bash
# Create in scripts/tests/
touch scripts/tests/test_new_component.sh
chmod +x scripts/tests/test_new_component.sh
```

### 2. Follow Template

```bash
#!/bin/bash
# Functional Test: Your Component
# Description of what it tests
set -e

# Load environment
source .env

# Verify required vars
if [ -z "$REQUIRED_VAR" ]; then
    echo "❌ Missing REQUIRED_VAR"
    exit 1
fi

# Run tests
echo "Testing component..."

# Test 1
echo -n "  Test 1... "
if your_test_command; then
    echo "✅ Passed"
else
    echo "❌ Failed"
    exit 1
fi

echo "✅ All tests passed"
exit 0
```

### 3. Add to Master Suite

Edit `test_system_functionality.sh`:

```bash
run_test "Your Component" "scripts/tests/test_new_component.sh" || true
```

## Performance Benchmarks

Expected runtimes on typical hardware:

| Script | Runtime | Checks | Database Queries |
|--------|---------|--------|------------------|
| `validate-env.sh` | ~1s | 10-15 | 0 |
| `health_check.sh` | ~3-5s | 6-8 | 5-10 |
| `verify_database_system.sh` | ~5-10s | 70 | 40-50 |
| `test_system_functionality.sh` | ~15-30s | 100+ | 50-100 |

## Environment Requirements

All scripts require:
- `.env` file in project root
- `bash` version ≥4.0
- `psql` PostgreSQL client
- Network access to 192.168.86.200 (for database/Kafka)

Optional tools:
- `docker` CLI (for service management)
- `jq` (for JSON parsing)
- `kafkacat` or `kcat` (for Kafka diagnostics)

## Related Documentation

- **Main Documentation**: `../CLAUDE.md`
- **Database Guide**: `DATABASE_VERIFICATION_GUIDE.md`
- **Infrastructure Guide**: `~/.claude/CLAUDE.md`
- **Observability**: `../docs/observability/AGENT_TRACEABILITY.md`
- **Configuration**: `../config/README.md`

## Maintenance Schedule

### Daily
- Run `health_check.sh` to monitor service status
- Check `/tmp/health_check_history.log` for trends

### Before Major Changes
- Run `verify_database_system.sh` to establish baseline
- Run `test_system_functionality.sh` for full verification

### After Deployments
- Run `test_system_functionality.sh` immediately
- Monitor for 24h with `health_check.sh`

### Weekly
- Review health check history for patterns
- Update verification scripts if new tables/views added
- Validate environment configuration

## Getting Help

If tests fail:
1. **Read the error output** - scripts provide detailed failure messages
2. **Check logs** - `/tmp/` files contain additional details
3. **Review documentation** - see links above
4. **Manual verification** - use `psql`, `docker`, `curl` to debug
5. **Service restart** - `docker restart <service-name>` often resolves issues

## Version History

- **v2.0** (2025-11-09): Added comprehensive database verification script (70 checks)
- **v1.5** (2025-11): Integrated with master test suite
- **v1.0** (2025-10): Initial health check and functional tests
