===============================================================================
LABORATORY EXERCISE 09: SRE GOLDEN SIGNALS (LETS/RED) MONITORING
STEP-BY-STEP EXECUTION GUIDE
===============================================================================

ENVIRONMENT SETUP:
------------------
1. Extract all 5 files from this archive into a dedicated directory.
2. Open a terminal in that directory and launch the environment:
      docker compose up -d --build
3. Verify both containers are running (`exam-portal` and `prometheus`):
      docker compose ps


-------------------------------------------------------------------------------
TASK 1: Measure Timing Metrics (DNS, TCP, TLS, TTFB, Total) using curl (2 Marks)
-------------------------------------------------------------------------------
Run the following curl command in your terminal to capture precise HTTP 
latency breakdowns for the exam portal:

  curl -o /dev/null -s -w '
  ===================================================
        SRE HTTP Response Timing Breakdown
  ===================================================
    DNS Lookup Time    (%{time_namelookup}s) : %{time_namelookup} sec
    TCP Connect Time   (%{time_connect}s)    : %{time_connect} sec
    TLS Handshake Time (%{time_appconnect}s) : %{time_appconnect} sec
    Time To First Byte (%{time_starttransfer}s) : %{time_starttransfer} sec
    Total Request Time (%{time_total}s)        : %{time_total} sec
  ===================================================
  ' http://localhost:5000/exam/login

WHAT TO RECORD FOR YOUR REPORT:
  - DNS Lookup (time_namelookup): Host domain resolution latency.
  - TCP Connect (time_connect): 3-Way Handshake establishment time.
  - TLS Handshake (time_appconnect): SSL/TLS handshake time (0 for plain HTTP).
  - TTFB (time_starttransfer): Waiting time until server sends first response byte.
  - Total Time (time_total): Complete end-to-end request duration.


-------------------------------------------------------------------------------
TASK 2: Calculate Request Rate & Error Rate in Prometheus (3 Marks)
-------------------------------------------------------------------------------
Step A: Generate Sample Traffic
1. Reset portal to normal operation:
      curl http://localhost:5000/simulate/normal

2. Send 30 normal login requests:
      for i in {1..30}; do curl -s http://localhost:5000/exam/login > /dev/null; done

3. Enable degraded mode and send 30 answer script submissions:
      curl http://localhost:5000/simulate/degradation
      for i in {1..30}; do curl -s -X POST http://localhost:5000/exam/submit > /dev/null; done

Step B: Run PromQL Queries in Prometheus UI
1. Open Prometheus Web Console: http://localhost:9090

2. Calculate Total Request Rate (req/sec over 1m window):
      sum(rate(http_requests_total[1m]))

3. Calculate Overall HTTP Error Rate Percentage (% 5xx status codes):
      (sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))) * 100

4. Calculate Error Rate per Endpoint:
      sum(rate(http_requests_total{status=~"5.."}[1m])) by (endpoint)


-------------------------------------------------------------------------------
TASK 3: Analyze LETS Golden Signals & Identify Bottleneck (3 Marks)
-------------------------------------------------------------------------------
Step A: Trigger Degradation Mode
  curl http://localhost:5000/simulate/degradation

Step B: Send 50 concurrent submission requests
  for i in {1..50}; do curl -s -X POST http://localhost:5000/exam/submit > /dev/null & done; wait

Step C: Observe the 4 Golden Signals in Prometheus (http://localhost:9090):
  1. LATENCY (TTFB / Duration 95th Percentile):
        histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[2m])) by (le))
     Observation: Spikes from ~0.15s to 3.85s.

  2. ERRORS (5xx Failures):
        sum(rate(http_requests_total{status="503"}[2m]))
     Observation: ~25% of submission requests fail with HTTP 503.

  3. TRAFFIC (Request Volume):
        sum(rate(http_requests_total[2m]))
     Observation: Request rate surges during submission deadline window.

  4. SATURATION (Worker Pool / CPU):
        exam_portal_active_workers
     Observation: Active thread worker pool reaches 95/100 limit (Capacity limit reached).

Step D: Bottleneck Identification Conclusion
  - Primary Bottleneck: Database Write Lock / Thread Pool Saturaion.
  - Explanation: Concurrent submission transactions exhaust worker thread connections, 
    causing database queue build-up, severe TTFB inflation, and HTTP 503 timeouts.


-------------------------------------------------------------------------------
TASK 4: Recommend SLI, SLO and Alert Condition with Justification (2 Marks)
-------------------------------------------------------------------------------
1. Service Level Indicator (SLI):
   - Latency SLI: Ratio of valid requests completed with TTFB < 500ms.
     PromQL:
     sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m])) / sum(rate(http_requests_total[5m])) * 100

   - Availability SLI: Ratio of HTTP requests returning non-5xx status codes.
     PromQL:
     (1 - (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) ) * 100

2. Service Level Objective (SLO):
   - Latency SLO: 99% of exam submission requests must respond in < 500ms over a rolling 30-day window.
   - Availability SLO: 99.9% of all portal requests must succeed (non-5xx) over a rolling 30-day window.

3. Alert Condition (Prometheus Rule Definition):
   - Critical Alert Rule:
       alert: ExamPortalHighErrorRate
       expr: (sum(rate(http_requests_total{status=~"5.."}[2m])) / sum(rate(http_requests_total[2m]))) > 0.05
       for: 1m
       labels:
         severity: critical
       annotations:
         summary: "Exam Portal HTTP Error Rate exceeds 5%"

4. Justification:
   - During university exam submission windows, high error rates lead to unsubmitted answer 
     scripts. Alerting on a 5% error rate sustained for 1 minute allows SREs to trigger auto-scaling 
     or queue-shedding before burning through the monthly Error Budget.
===============================================================================
