Metadata-Version: 2.4
Name: aice-observability
Version: 0.1.0
Summary: Unified OpenTelemetry-based observability layer for internal Python services
Author-email: Arnold Opiyo <arnoldopiyo@adanianlabs.io>
Keywords: metrics,observability,opentelemetry,telemetry,tracing
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27.0
Requires-Dist: opentelemetry-instrumentation-django>=0.64b0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.48b0
Requires-Dist: opentelemetry-instrumentation-flask>=0.64b0
Requires-Dist: opentelemetry-instrumentation-httpx>=0.48b0
Requires-Dist: opentelemetry-instrumentation-logging>=0.48b0
Requires-Dist: opentelemetry-instrumentation-psycopg2>=0.64b0
Requires-Dist: opentelemetry-instrumentation-redis>=0.48b0
Requires-Dist: opentelemetry-instrumentation-sqlalchemy>=0.48b0
Requires-Dist: opentelemetry-instrumentation-system-metrics>=0.48b0
Requires-Dist: opentelemetry-sdk>=1.27.0
Provides-Extra: dev
Requires-Dist: fastapi>=0.100.0; extra == 'dev'
Requires-Dist: httpx>=0.24.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.3.0; extra == 'dev'
Requires-Dist: redis>=5.0.0; extra == 'dev'
Requires-Dist: sqlalchemy>=2.0.0; extra == 'dev'
Description-Content-Type: text/markdown

Unified OpenTelemetry bootstrap for internal Python services. The objective is to provide every service with a consistent operational view without requiring teams to build dashboards from scratch.

## Usage

```python
from fastapi import FastAPI
from observability import Observability

app = FastAPI()

obs = (
    Observability()
    .service("data-quality-service", version="2.0.0")
    .start()                        # must come before instrument_*() calls
    .instrument_fastapi(app)
    .instrument_sqlalchemy(engine)
)
```

`.start()` must be called before any `instrument_*()` method. OpenTelemetry's
instrumentation libraries bind to whatever provider is globally registered
at the moment they run; calling them before `.start()` would silently do
nothing.

httpx is instrumented automatically inside `.start()`, since
they need no app-specific object.

## Configuration (Environment Variables)

| Variable | Required | Default | Description |
|---|---|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Yes, unless all telemetry is disabled | none | OTLP HTTP endpoint used to export traces, metrics, and logs. Examples: `http://192.168.100.254:4318` (Collector), `https://otlp-gateway-prod-us-central-0.grafana.net/otlp` (Grafana Cloud). |
| `OTEL_EXPORTER_OTLP_HEADERS` | No | none | Optional authentication headers in `key=value,key=value` format. Required by many hosted OTLP providers. |
| `OTEL_SERVICE_NAME` | Yes | none | Logical service name reported to the observability backend (for example `dq-service`). |
| `OTEL_ENVIRONMENT` | No | `development` | Deployment environment such as `development`, `staging`, or `production`. |
| `OTEL_TRACES_ENABLED` | No | `true` | Enable or disable distributed tracing. |
| `OTEL_METRICS_ENABLED` | No | `true` | Enable or disable metrics collection. |
| `OTEL_LOGS_ENABLED` | No | `true` | Enable or disable log export. |
| `OTEL_LOG_LEVEL` | No | `INFO` | Minimum application log level captured by the logging pipeline. |

### Example: Self-hosted OpenTelemetry Collector

```env
OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.100.254:4318
OTEL_SERVICE_NAME=dq-service
OTEL_ENVIRONMENT=development
OTEL_TRACES_ENABLED=true
OTEL_METRICS_ENABLED=true
OTEL_LOGS_ENABLED=true
```


# Standard Observability Dashboards


The dashboards are designed around operational questions rather than telemetry types. Instead of exposing every metric collected by OpenTelemetry, each dashboard focuses on answering a specific set of questions that engineers, tech leads, and platform teams routinely ask.

Version 1 of the observability platform includes the following dashboards:

1. Service Overview
2. Infrastructure
3. HTTP/API
4. Database (conditional)
5. External Services (HTTP Client)
6. Logs
7. Traces

---

# 1. Service Overview Dashboard

## Purpose

The Service Overview dashboard is the primary landing page for every service.

It provides a high level summary of service health, traffic, latency, failures, and operational status.

This dashboard should answer the following questions within a few seconds:

* Is the service healthy?
* Is it currently receiving traffic?
* Has latency increased?
* Are users experiencing failures?
* Should I investigate further?

## Target Audience

* Tech Leads
* Backend Engineers
* DevOps Engineers
* SREs
* Engineering Managers

## Panels

### Availability

Displays service uptime and overall availability.

Purpose:

Quickly determine whether the service is operational.

---

### Request Rate (Requests Per Second)

Displays incoming request volume.

Purpose:

Identify traffic spikes, traffic drops, and overall load.

---

### Active Requests

Shows the number of requests currently being processed.

Purpose:

Detect request buildup and possible bottlenecks.

---

### Error Rate

Percentage and total number of failed requests.

Purpose:

Quickly identify service degradation.

---

### Average Response Time

Average latency across all requests.

Purpose:

Monitor overall responsiveness.

---

### P95 Latency

95% of requests complete faster than this value.

Purpose:

Represents the experience of almost all users.

---

### P99 Latency

99% of requests complete faster than this value.

Purpose:

Highlights long-tail latency issues.

---

### HTTP Status Code Distribution

Breakdown of

* 2xx
* 3xx
* 4xx
* 5xx

Purpose:

Identify client errors versus server failures.

---

### Top Failing Endpoints

Endpoints with the highest error count.

Purpose:

Quickly identify problematic APIs.

---

### Slowest Endpoints

Endpoints ranked by average latency.

Purpose:

Locate performance bottlenecks.

---

### Service Metadata

Displays

* Service Name
* Version
* Environment
* Instance Count
* Last Restart Time

Purpose:

Provide deployment context during incident investigations.

---

### Quick Navigation

Links directly to

* Logs Dashboard
* Traces Dashboard
* Infrastructure Dashboard

Purpose:

Reduce investigation time.

---

# 2. Infrastructure Dashboard

## Purpose

The Infrastructure dashboard monitors resource utilization of the application process and host.

It helps determine whether performance issues originate from application logic or infrastructure limitations.

## Target Audience

* Platform Engineers
* DevOps Engineers
* SREs

## Panels

### CPU Usage

CPU utilization over time.

---

### Memory Usage

Current memory consumption.

---

### Memory Utilization

Memory usage as a percentage of available memory.

---

### Disk Usage

Disk space utilization.

---

### Disk I/O

Read and write throughput.

---

### Network Throughput

Incoming and outgoing traffic.

---

### File Descriptors

Number of open file descriptors.

---

### Thread Count

Active application threads.

---

### Garbage Collection Activity

Displays

* Collection frequency
* Objects collected
* GC pause duration

Purpose:

Identify memory pressure.

---

### Process Restarts

Detect unexpected application restarts.

---

# 3. HTTP/API Dashboard

## Purpose

Provides endpoint-level visibility into API behaviour.

Rather than viewing the service as a whole, this dashboard focuses on individual routes.

## Target Audience

* Backend Engineers
* API Owners

## Panels

### Requests Per Second by Endpoint

---

### Average Latency by Endpoint

---

### P95 Latency by Endpoint

---

### P99 Latency by Endpoint

---

### Error Rate by Endpoint

---

### Slowest Endpoints

---

### Largest Responses

---

### Largest Requests

---

### Request Duration Heatmap

Visualizes latency distribution across all requests.

---

### Endpoint Ranking

Ranks endpoints by

* Traffic
* Errors
* Latency

---

# 4. Database Dashboard

## Purpose

Provides visibility into database interactions.

This dashboard is displayed only when database instrumentation is enabled.

Supported instrumentations include

* SQLAlchemy
* psycopg2

## Target Audience

* Backend Engineers
* Database Administrators

## Panels

### Queries Per Second

---

### Average Query Duration

---

### Slowest Queries

---

### Database Error Rate

---

### Connection Pool Utilization

---

### Connection Pool Wait Time

---

### Pool Saturation

---

### Failed Queries

---

### Transaction Duration

---

### Database Retries

---

## Conditional Display

Visible only if one of the following instrumentations is enabled.

* SQLAlchemy
* psycopg2

---

# 5. External Services Dashboard

## Purpose

Provides visibility into outbound HTTP requests.

This dashboard helps identify slow or failing third-party services.

The dashboard is powered by HTTPX instrumentation.

## Target Audience

* Backend Engineers
* Tech Leads

## Panels

### Outbound Requests Per Second

---

### Average Outbound Latency

---

### P95 Outbound Latency

---

### Error Rate by Remote Service

---

### Timeout Count

---

### Retry Count

---

### Top Slowest Remote Hosts

---

### Request Distribution by Host

---

### External Dependency Health

Summarizes

* Success Rate
* Failure Rate
* Average Latency

---

# 6. Logs Dashboard

## Purpose

Provides centralized log exploration through Grafana and Loki.

The dashboard is intended for investigation after metrics indicate abnormal behaviour.

## Target Audience

* All Engineers

## Panels

### Live Logs

---

### Error Logs

---

### Warning Logs

---

### Log Volume Over Time

---

### Log Levels

Distribution of

* DEBUG
* INFO
* WARNING
* ERROR
* CRITICAL

---

### Service Filter

---

### Search

Supports free text search.

---

### Correlation Links

Jump directly to related traces.

---

# 7. Traces Dashboard

## Purpose

Provides distributed tracing through Jaeger.

The dashboard allows engineers to follow requests across services.

## Target Audience

* Backend Engineers
* Platform Engineers
* SREs

## Panels

### Recent Traces

---

### Slowest Traces

---

### Error Traces

---

### Trace Duration Distribution

---

### Service Dependency Graph

---

### Span Timeline

---

### Failed Spans

---

### Trace Search

Supports filtering by

* Service
* Endpoint
* Duration
* Status

---

### Correlated Logs

Jump directly to matching log entries.

---

# Dashboard Design Principles

Every dashboard should follow these principles.

## Minimal by Default

Dashboards should expose only the information required for operational decisions.

Avoid displaying every metric collected by OpenTelemetry.

---

## Progressive Investigation

Dashboards should naturally lead the user from high level health to detailed diagnostics.

Typical investigation flow:

Service Overview

↓

HTTP/API

↓

Database or External Services

↓

Logs

↓

Traces

---

## Consistency

All services should expose the same dashboard structure.

This reduces cognitive load when switching between projects.

---

## Conditional Sections

Dashboards should only display panels relevant to the service.

Examples:

A service without Redis should not display Redis metrics.

A service without SQLAlchemy should not display SQL metrics.

A service without HTTP clients should not display outbound request metrics.

---

## Standard Labels

All dashboards should support filtering using common resource attributes.

Examples include:

* Service Name
* Environment
* Version
* Instance
* Endpoint
* HTTP Method
* Database System
* Remote Host

---

# Future Dashboard Roadmap

The following dashboards are planned for future releases.

* Redis Dashboard
* Background Jobs Dashboard
* AI/LLM Dashboard
* Cache Dashboard
* Storage Dashboard
* Security Dashboard
* Business Metrics Dashboard


