===============================================================================
  TUTUM CICD ANSIBLE COLLECTION - VERSION 1.8.9 SUMMARY
===============================================================================

Release Date: 2026-01-27
Type: Minor Release (New Features + Critical Bug Fixes)
Backward Compatible: YES ✓

===============================================================================
  🎯 KEY HIGHLIGHTS
===============================================================================

1. OpenShift PostgreSQL Support
   - Fixed critical CrashLoopBackOff on OpenShift restricted-v2 SCC
   - Auto-detection and support for OpenShift/sclorg PostgreSQL images
   - Conditional environment variables and volume paths

2. Podman Support
   - Enterprise container runtime support for offline bundle creation
   - Auto-detection with fallback to Docker
   - Essential for RHEL, Oracle Linux, and air-gapped environments

3. Kubectl Wrapper Support
   - Native support for oc, microk8s kubectl, k3s kubectl
   - KUBECONFIG path handling for multi-cluster environments
   - Consistent support across all K8s roles

4. Manual CRD Installation
   - Enables operator deployment without cluster-admin permissions
   - Separate CRD manifest generation
   - Namespace-scoped operator with ClusterRole for CRD

5. Enhanced Operator Webhooks
   - Real-time certificate change notifications
   - Webhook endpoint: /webhook/certificate-change (port 8082)
   - Improved integration with tutum-engine

===============================================================================
  🐛 CRITICAL BUG FIXES
===============================================================================

PostgreSQL CrashLoopBackOff on OpenShift
-----------------------------------------
Problem:
  - Standard postgres image crashes on OpenShift with arbitrary UID
  - Restricted-v2 SCC assigns random UIDs (e.g., 1000920000)
  - Container exits immediately before probes execute

Solution:
  - Auto-detection of OpenShift environments
  - OpenShift-compatible image: quay.io/sclorg/postgresql-16-c9s
  - Conditional environment variables: POSTGRES_* → POSTGRESQL_*
  - Proper data paths: /var/lib/postgresql/data → /var/lib/pgsql/data
  - Adjusted probe timings for OpenShift images

Impact:
  ✓ PostgreSQL embedded mode now works on OpenShift
  ✓ Supports both standard K8s and OpenShift deployments
  ✓ Automatic configuration based on distribution

===============================================================================
  ✨ NEW FEATURES - DETAILED
===============================================================================

1. OPENSHIFT POSTGRESQL SUPPORT
--------------------------------

Automatic Configuration:
  k8s_distribution: openshift → Uses quay.io/sclorg/postgresql-16-c9s
  k8s_distribution: standard  → Uses postgres:15-alpine

Manual Override:
  k8s_tutum_postgres_image: quay.io/sclorg/postgresql-16-c9s
  k8s_tutum_postgres_version: latest

Image Type Auto-Detection:
  - Detects 'sclorg', 'rhscl', 'centos/postgresql' in image name
  - Automatically selects correct environment variables
  - Configures proper volume paths and probe timings

Environment Variables:
  Standard:  POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB
  OpenShift: POSTGRESQL_USER, POSTGRESQL_PASSWORD, POSTGRESQL_DATABASE

Data Paths:
  Standard:  /var/lib/postgresql/data (with subPath: pgdata)
  OpenShift: /var/lib/pgsql/data (no subPath)

Probe Timings:
  Standard:  initialDelaySeconds: 5
  OpenShift: initialDelaySeconds: 15 (slower startup)

Affected Roles:
  - k8s.tutum_postgres (runtime)
  - k8s.tutum_init (manifest generation)

2. PODMAN SUPPORT FOR OFFLINE BUNDLES
--------------------------------------

Configuration:
  k8s_tutum_offline_container_runtime: docker   # Default
  k8s_tutum_offline_container_runtime: podman   # Explicit
  k8s_tutum_offline_container_runtime: auto     # Auto-detect

Auto-Detection Logic:
  1. Check if Podman is available (command -v podman)
  2. If yes → Use Podman
  3. If no → Check if Docker is available
  4. If yes → Use Docker
  5. If neither → Fail with clear error message

Podman Environments:
  - OpenShift (always uses Podman)
  - RHEL 8+, Oracle Linux 8+
  - Rocky Linux, AlmaLinux, Fedora
  - Air-gapped corporate environments

Script Updates:
  - save-images.sh: Dynamic runtime variable
  - load-images.sh: Runtime validation + fallback logic
  - Both scripts test runtime availability before execution

Benefits:
  ✓ Works in Docker-less environments
  ✓ Better SELinux integration (RHEL/Fedora)
  ✓ Rootless by default
  ✓ No daemon process required

Affected Role:
  - k8s.tutum_offline_bundle

3. KUBECTL WRAPPER SUPPORT
---------------------------

Supported Wrappers:
  - kubectl          (Standard Kubernetes)
  - oc               (OpenShift)
  - microk8s kubectl (MicroK8s)
  - k3s kubectl      (k3s/Lightweight K8s)
  - Custom paths     (/snap/bin/microk8s kubectl)

Configuration Variables:
  k8s_kubectl_command: kubectl                    # Default
  k8s_kubectl_command: oc                         # OpenShift
  k8s_kubectl_command: "microk8s kubectl"         # MicroK8s
  k8s_kubectl_command: "/snap/bin/k3s kubectl"    # k3s with path

  k8s_kubectl_kubeconfig: ""                      # Default (uses ~/.kube/config)
  k8s_kubectl_kubeconfig: /etc/rancher/k3s/k3s.yaml
  k8s_kubectl_kubeconfig: ~/.kube/config-production

Auto-Detection (tutum_init):
  openshift → oc
  microk8s  → microk8s kubectl
  k3s       → k3s kubectl
  standard  → kubectl

Implementation:
  - Variable: k8s_kubectl = "{{ k8s_kubectl_command }}"
  - Environment block in all kubectl tasks:
    environment:
      KUBECONFIG: "{{ k8s_kubectl_kubeconfig if k8s_kubectl_kubeconfig | length > 0 else omit }}"

Affected Roles (8 roles updated):
  - k8s.tutum_engine
  - k8s.tutum_postgres
  - k8s.tutum_csi
  - k8s.tutum_operator
  - k8s.tutum_cnpg
  - k8s.tutum_certificate
  - k8s.tutum_manifests
  - k8s.tutum_init

Documentation:
  - KUBECTL-WRAPPER-GUIDE.md     (User guide)
  - KUBECTL-MIGRATION-EXAMPLE.md (Developer guide)

4. MANUAL CRD INSTALLATION
---------------------------

Use Case:
  Enterprise environments where users lack cluster-admin permissions
  for CustomResourceDefinition installation.

Configuration:
  k8s_tutum_operator_skip_crd_install: false      # Default (auto-install)
  k8s_tutum_operator_skip_crd_install: true       # Manual installation
  k8s_tutum_operator_crd_manifest_path: ./crd-for-admin.yml
  k8s_tutum_operator_watch_namespaces: "production,staging"

Workflow:
  1. User generates CRD manifest (without installing):
     ansible-playbook ... -e k8s_tutum_operator_skip_crd_install=true

  2. Manifest saved to: k8s_tutum_operator_crd_manifest_path

  3. Admin installs CRD (requires cluster-admin):
     kubectl apply -f crd-for-admin.yml

  4. Operator deployed with namespace-scoped permissions:
     - Role/RoleBinding (not ClusterRole/ClusterRoleBinding)
     - Watches only specified namespaces
     - No cluster-admin required

RBAC Structure:
  Cluster-wide:
    - CustomResourceDefinition (tutumcertificates.tutum.io)
    - ClusterRole for CRD operations
    - ClusterRoleBinding for operator ServiceAccount

  Namespace-scoped:
    - ServiceAccount (tutum-operator)
    - Role (secrets, configmaps, events)
    - RoleBinding

Benefits:
  ✓ Works in restricted environments (OpenShift, enterprise K8s)
  ✓ Clear separation: CRD (admin) vs Operator (user)
  ✓ Multi-namespace support with selective watching
  ✓ Backward compatible (auto-install by default)

Affected Role:
  - k8s.tutum_operator

5. WEBHOOK INTEGRATION
-----------------------

Feature: Real-time certificate change notifications

Architecture:
  tutum-engine → (HTTP POST) → tutum-operator-webhook:8082/webhook/certificate-change

Configuration (tutum-engine):
  WEBHOOK_ENABLED: "true"
  WEBHOOK_ENDPOINTS: "http://tutum-operator-webhook.tutum-system.svc:8082/webhook/certificate-change"

Service Definition:
  Name: tutum-operator-webhook
  Namespace: tutum-system
  Port: 8082
  Target: tutum-operator pod (webhook container port)

REST Method: POST
Endpoint: /webhook/certificate-change

Benefit:
  ✓ Immediate notification when certificates change
  ✓ Reduces sync latency (vs periodic polling)
  ✓ More efficient resource usage

Affected Roles:
  - k8s.tutum_engine (ConfigMap with webhook config)
  - k8s.tutum_operator (Deployment with webhook service)

===============================================================================
  📝 MODIFIED FILES (Summary)
===============================================================================

Roles Updated: 8 roles
  k8s.tutum_postgres        PostgreSQL OpenShift support
  k8s.tutum_init            PostgreSQL + kubectl wrapper config
  k8s.tutum_offline_bundle  Podman support
  k8s.tutum_operator        Manual CRD + kubectl wrapper
  k8s.tutum_engine          Kubectl wrapper
  k8s.tutum_csi             Kubectl wrapper
  k8s.tutum_cnpg            Kubectl wrapper
  k8s.tutum_certificate     Kubectl wrapper
  k8s.tutum_manifests       Kubectl wrapper

Key File Changes:
  defaults/main.yml         Added kubectl/podman/postgres configs (all roles)
  templates/deployment.yml  Conditional PostgreSQL paths/probes
  templates/secret.yml      Conditional PostgreSQL env vars
  tasks/*.yml               All kubectl commands updated with wrapper support
  templates/*.sh.j2         Dynamic container runtime selection

New Documentation:
  RELEASE-1.8.9.md          This release notes
  KUBECTL-WRAPPER-GUIDE.md  Kubectl wrapper user guide
  KUBECTL-MIGRATION-EXAMPLE.md  Developer migration guide
  VERSION-1.8.9-SUMMARY.txt This file

Updated Files:
  galaxy.yml                Version: 1.8.9
  CHANGELOG.md              Added 1.8.9 entry
  README.md files           Updated with new parameters

===============================================================================
  🚀 USAGE EXAMPLES
===============================================================================

OpenShift with Embedded PostgreSQL:
------------------------------------
ansible-playbook playbooks/install-tutum-k8s.yml \
  -e k8s_distribution=openshift \
  -e k8s_kubectl_command=oc \
  -e k8s_tutum_db_mode=embedded

✓ Auto-selects OpenShift PostgreSQL image
✓ Uses oc CLI wrapper

RHEL with Podman (Offline Bundle):
-----------------------------------
ansible-playbook playbooks/create-offline-bundle.yml \
  -e k8s_tutum_offline_container_runtime=podman \
  -e k8s_tutum_offline_output_dir=/tmp/tutum-offline

✓ Uses Podman instead of Docker

MicroK8s Installation:
----------------------
ansible-playbook playbooks/install-tutum-k8s.yml \
  -e k8s_distribution=microk8s \
  -e 'k8s_kubectl_command="microk8s kubectl"'

✓ Uses MicroK8s kubectl wrapper

k3s with Custom KUBECONFIG:
---------------------------
ansible-playbook playbooks/install-tutum-k8s.yml \
  -e k8s_distribution=k3s \
  -e 'k8s_kubectl_command="k3s kubectl"' \
  -e k8s_kubectl_kubeconfig=/etc/rancher/k3s/k3s.yaml

✓ Custom KUBECONFIG path
✓ k3s kubectl wrapper

Operator without Cluster-Admin (Manual CRD):
---------------------------------------------
ansible-playbook playbooks/install-tutum-operator.yml \
  -e k8s_tutum_operator_skip_crd_install=true \
  -e k8s_tutum_operator_crd_manifest_path=./crd-for-admin.yml \
  -e k8s_tutum_operator_watch_namespaces="production,staging"

# Admin installs CRD separately:
kubectl apply -f crd-for-admin.yml

✓ Namespace-scoped permissions
✓ No cluster-admin required

===============================================================================
  ✅ VERIFICATION COMMANDS
===============================================================================

Check PostgreSQL on OpenShift:
-------------------------------
oc get pods -n tutum-system
oc describe pod tutum-postgres-xxx -n tutum-system
oc get deployment tutum-postgres -n tutum-system -o yaml | grep image:
# Expected: quay.io/sclorg/postgresql-16-c9s:latest

Check Podman Runtime:
---------------------
cat /tmp/tutum-offline/save-images.sh | grep RUNTIME
cat /tmp/tutum-offline/load-images.sh | grep RUNTIME
podman images | grep tutum

Check Kubectl Wrapper:
----------------------
cat /tmp/tutum-inventory.ini | grep k8s_kubectl
oc cluster-info  # OpenShift
microk8s kubectl get nodes  # MicroK8s

Check CRD and Operator:
-----------------------
kubectl get crd tutumcertificates.tutum.io
kubectl get clusterrole tutum-operator
kubectl get role tutum-operator -n tutum-system
kubectl get svc tutum-operator-webhook -n tutum-system

Check Webhook:
--------------
kubectl get endpoints tutum-operator-webhook -n tutum-system
kubectl logs -n tutum-system deployment/tutum-operator | grep webhook

===============================================================================
  🔄 BACKWARD COMPATIBILITY
===============================================================================

✓ 100% BACKWARD COMPATIBLE

- Existing playbooks work without modifications
- Default behavior unchanged:
  * Standard postgres image (postgres:15-alpine)
  * Docker container runtime
  * Standard kubectl command
  * Automatic CRD installation

- All new features are OPT-IN via configuration variables
- No breaking changes to roles or playbooks
- Inventory files from 1.8.x work without changes

Migration Notes:
  - OpenShift users: Add k8s_distribution=openshift
  - Podman users: Add k8s_tutum_offline_container_runtime=podman
  - Wrapper users: Add k8s_kubectl_command=<wrapper>
  - All optional with sensible defaults

===============================================================================
  📦 INSTALLATION
===============================================================================

Install from Ansible Galaxy:
----------------------------
ansible-galaxy collection install tutum_pro.cicd:1.8.9

Install from Git:
-----------------
ansible-galaxy collection install \
  git+https://github.com/tutum-pro/tutum-cicd.git,1.8.9

Update Existing Installation:
------------------------------
ansible-galaxy collection install tutum_pro.cicd:1.8.9 --force

Verify Installation:
--------------------
ansible-galaxy collection list | grep tutum_pro.cicd
# Expected: tutum_pro.cicd  1.8.9

===============================================================================
  🔒 SECURITY NOTES
===============================================================================

PostgreSQL Images:
  - OpenShift images from quay.io/sclorg (Red Hat maintained)
  - Standard images from Docker Hub (official repository)
  - Both follow security best practices

Podman Security:
  - Rootless by default when run as non-root
  - SELinux integration on RHEL/Fedora/CentOS
  - No daemon process required (more secure)

Kubectl Wrappers:
  - Always validate KUBECONFIG paths
  - Use service accounts for automation
  - Never store credentials in inventory files
  - Use Ansible Vault for sensitive data

Operator:
  - Runs as non-root (UID 65532)
  - Minimal permissions (namespace-scoped option)
  - Read-only root filesystem
  - Drops all capabilities

===============================================================================
  📊 RELEASE STATISTICS
===============================================================================

Files Modified:     25+
Files Added:        3
Lines Changed:      ~500
Roles Enhanced:     8
New Features:       5 major features
Bug Fixes:          1 critical fix
Documentation:      6 new/updated files

Affected Components:
  - PostgreSQL (embedded mode)
  - Offline bundle generator
  - Tutum Operator
  - All K8s roles (kubectl wrapper)

Supported Platforms Enhanced:
  - OpenShift 4.x
  - MicroK8s
  - k3s / RKE2
  - RHEL 8+, Oracle Linux 8+
  - Rocky Linux, AlmaLinux
  - Fedora
  - Air-gapped environments

===============================================================================
  📞 SUPPORT & RESOURCES
===============================================================================

Issues:        https://github.com/tutum-pro/tutum-cicd/issues
Documentation: https://www.tutum-pro.io
Repository:    https://github.com/tutum-pro/tutum-cicd

Documentation Files:
  - RELEASE-1.8.9.md              Detailed release notes
  - CHANGELOG.md                  Version history
  - KUBECTL-WRAPPER-GUIDE.md      Kubectl wrapper guide
  - KUBECTL-MIGRATION-EXAMPLE.md  Developer examples
  - README.md                     Main documentation
  - OPENSHIFT-RBAC-README.md      OpenShift RBAC (v1.8.8)

===============================================================================
  🎉 THANK YOU!
===============================================================================

This release represents significant progress in enterprise Kubernetes support,
particularly for OpenShift and restricted environments.

Special thanks to enterprise users who provided feedback on:
  - OpenShift compatibility issues
  - Podman requirements in corporate environments
  - Kubectl wrapper needs for different distributions
  - Security constraints in restricted environments

Upgrade today for enhanced OpenShift and enterprise Kubernetes support!

===============================================================================
