UXDL Docs
Observability

OpenTelemetry & Tempo

Distributed tracing implementation for AWS ECS using Grafana Tempo and S3.

This proposal details the implementation of end-to-end distributed tracing for applications running on AWS ECS. The initial phase focuses strictly on distributed tracing; metrics and log collection are designated for future phases.

Objective & Challenges

Objective

Implement distributed tracing across ECS services to:

  • Map request flows across microservices.
  • Identify latency bottlenecks.
  • Troubleshoot failures in real time.
  • Reduce Mean Time to Resolution (MTTR).

Current Challenges

  • Limited visibility into request flows between services.
  • Latency source identification is slow and difficult.
  • Manual log analysis is required to troubleshoot cross-service issues.
  • No unified end-to-end view of user requests.

Proposed Solution

We implement an OpenTelemetry-based tracing stack using a centralized collector and Grafana Tempo with Amazon S3 for long-term storage.

High-Level Architecture

Components

  1. OpenTelemetry SDK / Auto-Instrumentation
    • Integrates into each application service to automatically generate traces, record spans, and propagate trace context.
  2. OpenTelemetry Collector
    • A dedicated ECS service running as a sidecar or centralized daemon.
    • Receives traces, batches/processes telemetry, and exports data to Grafana Tempo.
  3. Grafana Tempo
    • High-scale, cost-effective distributed tracing backend.
  4. Amazon S3
    • Long-term storage backend for Tempo trace blocks, offering durability and low storage cost.
  5. Grafana UI
    • Visualization interface for searching traces, analyzing dependencies, and identifying slow queries/services.

Deployment Plan

Phase 1 — Infrastructure Setup

  1. Create Amazon S3 Bucket: Create an S3 bucket (e.g., telemetry-traces-prod) to store Tempo trace data.
  2. Deploy Grafana Tempo: Deploy Tempo as a private ECS service with backend storage configured to the S3 bucket.
  3. Deploy OpenTelemetry Collector: Deploy the collector as a dedicated ECS service with an OTLP receiver and Tempo exporter configured.
  4. Configure Grafana: Add Tempo as a datasource in Grafana and set up tracing dashboards.

Application Integration

Environment Variables

Configure each service with the following environment variables:

dotenv
OTEL_SERVICE_NAME=<service-name>
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_TRACES_EXPORTER=otlp

Instrumentation Strategy

Where supported, leverage auto-instrumentation to minimize code modification:

  • Java: Run with the OpenTelemetry Java Agent (-javaagent:opentelemetry-javaagent.jar).
  • Node.js: Use @opentelemetry/auto-instrumentations-node packages.
  • Python: Use opentelemetry-instrument CLI wrappers.

Trace Flow Example

When a client request travels through the stack:

The resulting trace tree visualizes the execution sequence and latency breakdown:

plaintext
Trace (Total: 420ms)
├── API Request (api-gateway) [420ms]
├── Authentication (auth-service) [80ms]
├── Business Logic (order-service) [310ms]
└── Database Query (order-service) [120ms]

Future Enhancements

After successfully establishing distributed tracing, the stack will expand to cover:

  • Phase 2 — Metrics: Collect OpenTelemetry metrics, exporting them to Prometheus/Grafana Mimir.
  • Phase 3 — Logs: Standardize structured logging and ship logs to Grafana Loki, correlating logs with traces.
  • Phase 4 — Unified Observability: Establish a single pane of glass in Grafana combining Traces, Metrics, and Logs.