OpenTelemetry & Axiom

This document outlines the implementation of end-to-end distributed tracing for applications running on AWS ECS using OpenTelemetry and Axiom.

Objective & Challenges

Objective

Establish distributed tracing with minimal operational overhead by utilizing Axiom as a fully managed observability backend.

Current Challenges

• Limited visibility into request flows between microservices.
• Difficult latency identification across cross-service boundaries.
• Manual log analysis required to troubleshoot incidents.
• No centralized trace view for requests spanning multiple services.

Proposed Solution

We implement a vendor-neutral OpenTelemetry instrumentation layer that exports traces to Axiom via a centralized OpenTelemetry Collector.

High-Level Architecture

Components

1. OpenTelemetry SDK
   • Integrated into each application service.
   • Responsible for generating traces, creating spans, propagating context, and capturing database/API operations.
2. OpenTelemetry Collector
   • Centralized ECS service that receives telemetry from applications, batches and processes traces, and exports them to Axiom.
3. Axiom
   • Serves as the centralized observability backend, providing managed trace storage, indexing, high-performance query execution, and distributed trace visualization.

Architecture Benefits

Reduced Infrastructure

Using Axiom simplifies the traditional OpenTelemetry tracing stack by removing several components:

Simplified Operations

By removing Tempo, S3, and Grafana from our infrastructure footprint, we eliminate the need to manage:

• Grafana deployments, dashboards, and access controls.
• Grafana Tempo instances, scaling rules, and write-paths.
• S3 storage backends, trace retention/compaction, and backups.
• Telemetry database scaling and maintenance overhead.

Deployment Plan

Phase 1 – Axiom Setup

• Create a dedicated Axiom organization.
• Initialize a tracing dataset (e.g., traces-prod).
• Generate ingest API tokens for the OTel Collector.

Phase 2 – OpenTelemetry Collector Deployment

Deploy the OpenTelemetry Collector as a dedicated ECS service. Configure the receiver and exporter:

• OTLP Receiver: Accepts OTLP traces from application services.
• Axiom Exporter: Configured with the Axiom API token and endpoint to forward traces.

Phase 3 – Application Integration

Configure each application service (such as api-gateway, auth-service, user-service, payment-service, notification-service) with:

OTEL_SERVICE_NAME=<service-name>
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Trace Flow Example

A single client request produces one distributed trace spanning every service it touches:

The resulting trace gives end-to-end request visibility:

Trace
├── HTTP Request
├── Authentication
├── Business Logic
├── Database Query
└── Response

Application Instrumentation Strategy

Auto-Instrumentation (Preferred)

Auto-instrumentation captures typical framework, HTTP, and DB spans with minimal code modification:

• Java: Java Agent injection.
• Node.js: Auto-instrumentation packages.
• Python: Wrapper scripts/middleware.

Manual Instrumentation

Apply manual tracing using the OpenTelemetry API for critical transactions and business workflows:

• Payment processing: Spans detailing gateway latency and payload statuses.
• User onboarding: Track user creation, profile caching, and confirmation email triggers.
• Order placement: Trace inventory reserves, transactional writes, and downstream messaging.

Expected Benefits

• Faster root cause analysis — quickly identify slow services, failed requests, database bottlenecks, and external dependency issues.
• Service dependency visibility — visualize communication between services (for example, API Gateway → Auth, User, and Payment services).
• Reduced troubleshooting time — engineers can determine where requests fail, which service is responsible, and how latency is distributed.
• Improved production visibility — real-time insight into application behavior across the ECS environment.

Future Enhancements

This phase focuses only on distributed tracing. Metrics and logs are added in later phases.

Risks & Considerations

SaaS Dependency

All observability data is stored in Axiom.

• Mitigation: We use standard, open-source OpenTelemetry SDKs and protocols. If needed, the backend can be swapped in the future (e.g., to Tempo or Datadog) without re-instrumenting application code.

Usage-Based Pricing

Ingestion costs depend on trace volume, retention period, and query usage.

• Mitigation: Start by instrumenting production-critical services only. Set parent-based trace sampling rates (e.g., 5-10% in production) to control ingestion scale, while maintaining 100% sampling in Alpha/Beta environments.

Recommendation

Implement distributed tracing using the OpenTelemetry SDK, OpenTelemetry Collector, and Axiom. This provides a modern, low-maintenance observability solution with minimal infrastructure while keeping instrumentation vendor-neutral — enabling rapid implementation, simplified operations, and a scalable foundation for metrics and logs in later phases.