A Complete Introductory Guide to OpenTelemetry | Better Stack Community (2024)

OpenTelemetry is an open-source observability framework that addresses thelimitations of existing telemetry agents by providing a unified, vendor-neutralapproach for collecting and exporting telemetry data.

At its core, OpenTelemetry facilitates the instrumentation of applications in avendor-agnostic manner, allowing for the subsequent analysis of collected datathrough backend tools like Prometheus, Jaeger, Zipkin, and others, according toyour preference.

The broad scope and open-source nature of OpenTelemetry can sometimes createconfusion and skepticism. This article seeks to clarify OpenTelemetry's role byexplaining its key features, practical benefits, and how it can significantlyimprove your observability strategy.

Let's get started!

What is telemetry data?

In simple terms, telemetry data is the information gathered from variousdeployments of software applications, systems, and devices for monitoring andanalysis purposes.

It is an essential aspect of understanding how systems operate, identifyingperformance issues, troubleshooting problems, and making informed decisionsabout optimization and resource allocation.

In the era of cloud-native and microservice architectures, where applicationsare complex and distributed, collecting and analysing telemetry data has becomeeven more critical for achieving full observabilityinto your systems.

The most common types of telemetry data include:

• Metrics: Numerical measurements that quantify system health orperformance, such as CPU usage, memory consumption, request latency, and errorrates.

• Traces: Records the path taken by a request as it moves through adistributed system, highlighting dependencies and bottlenecks.

• Logs: Records significant actions, errors, and other relevant informationthat help with understanding system behavior and troubleshooting issues.

• Events: Structured records containing contextual information about what ittook to complete a unit of work in an application (commonly a single HTTPtransaction).

• Profiles: Provides insights into resource usage (CPU, memory, etc.) withinthe context of code execution.

What is OpenTelemetry?

OpenTelemetry (often abbreviated as OTel) is an open-source observabilityframework and toolkit for generating, collecting, and exporting telemetry datafrom your software applications and infrastructure.

It emerged in 2019through a mergerof OpenTracing andOpenCensus. Each project had unique strengths but alsolimitations that hindered broader adoption. By combining their best featuresunder the Cloud Native Computing Foundation (CNCF), OpenTelemetry provides aunified, standardized framework for collecting all kinds of observabilitysignals, addressing the shortcomings of its predecessors.

At the time of writing, theCNCF development statisticsshow that OpenTelemetry is currently the 2nd most active CNCF project, onlysurpassed by Kubernetes.

What problem is OpenTelemetry aimed at solving?

OpenTelemetry aims to address the fragmentation and complexity of how telemetrydata is collected and processed in distributed systems. It seeks to replace themyriad of proprietary agents and formats with a unified, vendor-neutral, andopen-source standard for instrumenting applications, collecting signals (traces,metrics, and logs), and exporting them to various analysis backends.

Note that OpenTelemetry focuses solely on collecting and delivering telemetrydata, leaving the generation of actionable insights to dedicated analysis toolsand platforms.

Components of OpenTelemetry

The OpenTelemetry framework comprises ofseveral components thatwork together to capture and process telemetry data, which are outlined below:

1. API & SDK specification

The OpenTelemetry specificationdefines the standards, requirements, and expectations for implementingOpenTelemetry across different programming languages, frameworks, andenvironments. It is divided into three major sections:

1. API specification: This defines the data types and programming interfacesfor creating and manipulating telemetry data in different languages to ensureconsistency in how such data is generated and handled across various systems.

2. SDK specification: This defines the behavior and requirements for thelanguage-specific implementation of the OpenTelemetry API. SDKs handle taskslike sampling, context propagation, processing, and exporting telemetry data.It also enablesautomatic instrumentation throughintegrations and agents, which reduces the need for manual coding to capturemetrics and traces.

3. Data specification: This defines the OpenTelemetry Protocol (OTLP), avendor-agnostic protocol for transmitting telemetry data between differentcomponents of the OpenTelemetry ecosystem. It specifies the supportedtelemetry signals' data formats, semantic conventions, and communicationmechanisms to ensure consistency, making analyzing and correlating data fromdifferent sources easier.

2. Semantic conventions

OpenTelemetry semantic conventionsare standardized guidelines that define the naming and structure of attributesused to describe telemetry data. These conventions provide meaning to data whenproducing, collecting, and consuming it.

Some key aspects of OTel semantic conventions include:

1. Attribute naming: It provides a set of well-defined names for spanattributes, metrics, and other fields that represent common concepts,operations, and properties in different domains. For example,http.response.status_code represents the HTTP status code of a request,db.system denotes the database system being used, and exception.typeindicates the type of exception thrown.

2. Telemetry schemas: It defines the structure of attributes, their datatypes, and allowed values. This ensures telemetry data generated by differentcomponents can be seamlessly combined and correlated. It also allows theseschemas to evolve over time through versioning.

3. Collector

The OpenTelemetry Collector acts as an intermediary between your instrumentedapplications and the backend systems where you analyze and visualize your data.It is designed to be a standalone binary process (that can be run as a proxy orsidecar) that receives telemetry data in various formats, processes and filtersit then sends it to one or more configured observability backends.

The Collector is composed of the following key components that come together toform an observability pipeline:

1. Receiversfor ingesting telemetry data in different formats (OTLP, Prometheus, Jaeger,etc) and from various sources.

2. Processorsfor processing the ingested telemetry data by filtering, aggregating,enriching or transforming it.

3. Exportersfor delivering the processed data to one or more observability backends inwhatever format you desire.

4. Connectorsfor bridging different pipelines within the Collector, enabling seamless dataflow and transformation between them. They act as both an exporter for onepipeline and a receiver for another.

5. Extensionsfor providing additional functionality like health checks, profiling, andconfiguration management. These don't require direct access to telemetrydata.

It is also separated into two main GitHub repositories:

1. otel-collector: This is the coreproject focuses on the fundamental processing logic of the collector,specifically the handling and manipulation of OTLP data.

2. otel-collector-contrib:This project acts as a comprehensive repository of various integrations,including receivers for collecting telemetry data from different sources andexporters for sending data to diverse backends.

Due to the vast number of integrations, you are advised to create customotel-collector-contrib builds that include only the specific components youneed. This can be done through theOpenTelemetry Collector Builder tool.

4. Protocol (OTLP)

The OpenTelemetry Protocol(OTLP) is a vendor-neutral and open-source specification for how telemetry dataare encoded, transported, and delivered between different components within theOpenTelemetry ecosystem.

It enables seamless communication between various parts of your observabilitystack, regardless of the specific tools or platforms you're using. Thisflexibility prevents vendor lock-in and allows you to choose the tools that bestsuit your needs.

Note that OpenTelemetry also supports ingesting data in other protocols (such asZipkin, Prometheus, Jaeger, etc) with the appropriate receiver, and you canconvert data from one format to another to simplify integration with differentbackends.

5. Open Agent Management Protocol (OpAMP)

The Open Agent Management Protocolis an emerging open standard designed to manage large fleets of data collectionagents at scale. It was donated to the OpenTelemetry (OTel) project by Splunk in2022 and is currentlyunder active development withinthe OTel community.

OpAMP defines a network protocol for remote management of agents, includinginstances of the OpenTelemetry Collector, as well as vendor-specific agents thatimplement the OpAMP spec. This allows a centralized server (OpAMP control plane)to provide a "single pane of glass" view that monitors, configures, and updatesa large fleet of agents across a distributed environment.

6. Transformation Language (OTTL)

TheOpenTelemetry Transformation Languageis a powerful, flexible language designed to transform telemetry dataefficiently within the OpenTelemetry Collector. It provides a vendor-neutral wayto filter, transform, and modify data before it is exported to various analysisbackends.

It is still under active development as part of the otel-collector-contribproject, but it holds great potential for simplifying and standardizing theprocessing of telemetry data in observability pipelines.

7. Demo application

A microservice-based shopping siteshowcasing the capabilities of various OpenTelemetry features and language SDKs.It provides a practical example of how Otel can be used to instrument andobserve a distributed system in real-world scenarios.

What programming languages are supported?

OpenTelemetry supports a wide range of programming languages, making it a trulyuniversal observability framework. Here's a list of the officially supportedlanguage APIs and SDKs:

• Java
• JavaScript
• Python
• Go
• .NET
• C++
• Ruby
• PHP
• Erlang/Elixir
• Rust
• Swift

There are also community-supported SDKs and instrumentation libraries for otherlanguages, which can be found in theregistry.

Note that supported SDKs' maturity and feature set can vary across languages.While the core API is standardized, some language-specific implementations mighthavedifferences in featuresor stability levels.

You can find the official list of supported languages and their documentation onthe OpenTelemetry website.

OpenTelemetry signals and stability explained

Signals refer to the different types of telemetry data that the OpenTelemetryframework is designed to collect, process, and export. We're currently dealingwith three primary types of signals: distributed traces, metrics, and logs, withcontinuous profilingin early development as a fourth.

Each component, including the individual signal types, language-specific SDKs,and collector integrations is handled by a different group within theOpenTelemetry project, leading to a truly collaborative effort to develop andmaintain the framework.

In OpenTelemetry,"stability"refers to a specific stage in the maturity lifecycle of a component or signal.It could mean stability in the specification, semantic conventions, protocolrepresentation, language-specific SDKs, and the collector.

A component or signal deemed "stable" has a well-defined API, schema, andbehavior unlikely to undergo significant changes in future releases. Thisstability allows you to reliably build upon and integrate these components inyour production applications without concern for disruptive changes.

It's important to understand that stability in one area does not mean "stablefor everything". Always consult the official documentation toverify the latest stability status of anycomponents you plan to utilize in your projects.

Now, let's look at each major signal supported by OpenTelemetry and theirstability status:

1. Traces — stable

Distributed tracing within OpenTelemetryreached general availabilityin September 2021. This means the Tracing API, SDK, and Protocol specificationsare considered stable and suitable for production use.

At the time of writing, the OTel tracing implementation for allofficially supported languages isstable except for Rust, which is currently in beta.

2. Metrics — stable

OpenTelemetry metrics achieved general availabilityin 2021,signifying that its API, SDK, and Protocol specifications are production-readyfor various programming languages. That said, development for full SDK stabilityis still ongoing across the board.

3. Logs — stable

Thegeneral availabilityannouncement of OpenTelemetry logs at Kubecon North America's 2023 editionmarked a significant step towards wider adoption. It enables the OpenTelemetryCollector and APIs/SDKs to seamlessly capture, process, and export logs alongwith metrics and traces, making it an attractive solution for many organizationswho often start their observability journey with logs.

Creating a plan to adopt OpenTelemetry

Before embracing OpenTelemetry for your project, a thorough assessment of yourcurrent technology stack is necessary. Start by identifying the programminglanguages and frameworks powering your frontend and backend services. This willguide your selection of compatible client libraries and instrumentation agents.

Next, pinpoint the specific telemetry data (logs, metrics, or traces) you needto collect and their origins. Whether they're generated within your applicationor sourced from external systems like Kafka, Docker, or PostgreSQL,understanding this will direct your choice of receivers for the OpenTelemetryCollector.

If your existing code already generates telemetry data, determine whether itutilizes OpenCensus, OpenTracing, or another framework. OpenTelemetry isbackwards compatible with both OpenCensus and OpenTracing, which shouldeliminate the need for major initial code modifications.

However, to leverage the full potential of OpenTelemetry, a gradual migration isrecommended. If you rely on vendor-specific instrumentation, anticipate the needfor re-instrumentation using OpenTelemetry.

Finally, determine the destination of your telemetry data. Are you usingopen-source tools like Jaeger or Prometheus, a proprietary vendor solution, oreven a Kafka cluster for further processing? This decision will dictate theexporters you'll need within the OpenTelemetry Collector.

By mapping out your technology stack and identifying the relevant OpenTelemetrycomponents, you'll be well-prepared to evaluate their stability and readinessfor your project's specific needs.

Instrumenting an application with OpenTelemetry

Instrumentation with OpenTelemetry involves adding code manually or usingauto-instrumentation agents to generate telemetry data for each operationperformed in an application.

We'll focus on instrumenting a service for generating traces, where each serviceoperation emits one or more spans. Spans contain data on the service, operation,timing, context (trace/span IDs), and optional attributes.

The OpenTelemetry SDK propagates the span's context across services to establishcausal relationships, ensuring spans can be reconstructed into meaningful tracesfor analysis in backend tools.

Let's see these concepts in action by instrumenting a basic Node.js applicationwith OpenTelemetry and sending the generated data to Jaeger for analysis.

1. Begin with automatic instrumentation

Automatic instrumentationcan help jumpstart your observability journey by capturing data from manypopular libraries and frameworks without requiring any code changes. This meansyou can start collecting traces within minutes instead of doing everythingmanually.

Let's say you have a simple Fastify app like this:

import Fastify from 'fastify';const fastify = Fastify({ logger: false,});fastify.get('/', async function (request, reply) { const response = await fetch('https://icanhazdadjoke.com/', { headers: { Accept: 'application/json', }, }); const data = await response.json(); reply.send({ data });});const PORT = parseInt(process.env.PORT || '8080');fastify.listen({ port: PORT }, function (err, address) { if (err) { console.error(err); process.exit(1); } console.log(`Listening for requests on ${address}`);});

You can instrument it with OpenTelemetry through its Node.js SDK andauto-instrumentations package so that it automatically creates spans for eachincoming request.

Install the required packages first:

npm install @opentelemetry/sdk-node \ @opentelemetry/api \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-trace-node

Then set up the instrumentation in a different file:

import { NodeSDK } from '@opentelemetry/sdk-node';import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-node';import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';const sdk = new NodeSDK({ traceExporter: new ConsoleSpanExporter(), instrumentations: [getNodeAutoInstrumentations()],});sdk.start();

Finally, you must register the instrumentation before your application code likethis:

import './instrumentation.js';
import Fastify from 'fastify';. . .

Start the application and send a few requests to the / endpoint. You'll startseeing spans that track the lifetime of each request in the console:

. . .{ resource: { attributes: { . . . } }, traceId: '1bb070e6ff071ce5ae311695861ad5ae', parentId: 'fee8dce0965687ac', traceState: undefined, name: 'GET', id: 'dc00b1e8bb4e11c2', kind: 2, timestamp: 1715071474394000, duration: 524759.881, attributes: { 'http.request.method': 'GET', 'http.request.method_original': 'GET', 'url.full': 'https://icanhazdadjoke.com/', 'url.path': '/', 'url.query': '', 'url.scheme': 'https', 'server.address': 'icanhazdadjoke.com', 'server.port': 443, 'user_agent.original': 'node', 'network.peer.address': '2606:4700:3033::6815:420f', 'network.peer.port': 443, 'http.response.status_code': 200 }, status: { code: 0 }, events: [], links: []}{ resource: { attributes: { . . . } }, traceId: '1bb070e6ff071ce5ae311695861ad5ae', parentId: undefined, traceState: undefined, name: 'GET', id: 'fee8dce0965687ac', kind: 1, timestamp: 1715071474382000, duration: 544035.726, attributes: { 'http.url': 'http://localhost:8080/', 'http.host': 'localhost:8080', 'net.host.name': 'localhost', 'http.method': 'GET', 'http.scheme': 'http', 'http.target': '/', 'http.user_agent': 'curl/8.6.0', 'http.flavor': '1.1', 'net.transport': 'ip_tcp', 'net.host.ip': '::1', 'net.host.port': 8080, 'net.peer.ip': '::1', 'net.peer.port': 59792, 'http.status_code': 200, 'http.status_text': 'OK' }, status: { code: 0 }, events: [], links: []}

The trace for each request contains two spans: one for the request to the serverand the other for the GET request to icanhazdadjoke.com's API. Armed with suchdata, you can immediately pinpoint slowdowns within your services or theirdependencies.

Currently, automatic instrumentation is available for Java, .NET, Python,JavaScript, and PHP. Compiled languages like Go and Rust lack direct support,but automatic trace injection can still be achieved using external tools likeeBPF or service mesh technologies.

Let's look at how to add custom instrumentation next for even deeper insights.

2. Manually instrument your code

Automatic instrumentation gives you a solid foundation, but to truly understandthe inner workings of your system, you'll need custom instrumentation. This letsyou monitor the specific business logic that makes your application unique.

To get started, you need to identify the unit of work you'd like to track. Thiscould be function executions, cache interactions, background tasks, or otherinternal steps within a service

Assuming you have the following route in your application that calculates thespecified Fibonacci number:

. . .function fibonacci(n) { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2);}fastify.get('/fibonacci/:n', (request, reply) => { const n = parseInt(request.params.n, 10); const result = fibonacci(n); reply.send({ result });});. . .

You can use the following code to create a span for each Fibonacci computationlike this:

import './instrumentation.js';import Fastify from 'fastify';
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('fastify-app', '0.1.0');
. . .fastify.get('/fibonacci/:n', (request, reply) => { const n = parseInt(request.params.n, 10);
 const span = tracer.startSpan('calculate-fibonacci-number', {
 attributes: {
 'fibonacci.input': n,
 },
 });
 const result = fibonacci(n);
 span.setAttribute('fibonacci.result', result);
 span.end();
 reply.send({ result });});

Custom instrumentation starts with obtaining the tracer and creating a span forthe work you'd like to track. You can attach key/value pairs to the span toprovide more details about the operation that it's tracking. Once the operationis done, the span is finalized with span.end().

Such instrumentation will now capture spans detailing how long each Fibonaccicalculation takes, its input, and the result:

{ resource: { attributes: { . . . } }, traceId: '94acf0a34595230b72acbd473ca78617', parentId: '91888aca54a65286', traceState: undefined, name: 'calculate-fibonacci-number', id: '88ec3f8d2304a32e', kind: 0, timestamp: 1715076859034000, duration: 28.72, attributes: { 'fibonacci.input': 10, 'fibonacci.result': 55 }, status: { code: 0 }, events: [], links: []}

Up next, we'll explore how to visualize this collected data to troubleshootissues and optimize your application!

3. Export trace data to backend system

Now that you've generated all this helpful data, it's time to send it to abackend system for visualization and analysis. OpenTelemetry offers two mainexport methods:

1. The aforementioned OpenTelemetry collector which offersflexibility for data processing and routing to various backends.
2. A direct export from your application to one or more backends of your choice.

We'll use the second approach to export traces to Jaeger for simplicity. You canuse the following command tolaunch Jaeger in your local environment:

docker run --rm --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:1.57

Unable to find image 'jaegertracing/all-in-one:1.57' locally1.57: Pulling from jaegertracing/all-in-onea88dc8b54e91: Already exists1aad216be65d: Pull complete4b87021fa57f: Pull complete1c6e9aedbcb3: Pull complete7e4eba3a7c50: Pull completeDigest: sha256:8f165334f418ca53691ce358c19b4244226ed35c5d18408c5acf305af2065fb9Status: Downloaded newer image for jaegertracing/all-in-one:1.57. . .

Visit http://localhost:16686 to access the Jaeger UI. You should see:

OpenTelemetry includes exporter libraries for Node.js that allow you to pushrecorded spans directly to a consumer. In this case, you will push the generatedspans to your local Jaeger instance.

Start by installing the OpenTelemetry Collector exporter for Node.js with:

npm install --save @opentelemetry/exporter-trace-otlp-proto

Next, modify your instrumentation.js file as follows to configure theexporter:

import { NodeSDK } from '@opentelemetry/sdk-node';import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
const sdk = new NodeSDK({
 traceExporter: new OTLPTraceExporter(),
 instrumentations: [getNodeAutoInstrumentations()],});sdk.start();

Restart your application, and specify the OTEL_SERVICE_NAME environmentalvariable so that you can identify your application traces in Jaeger:

OTEL_SERVICE_NAME=fastify-app node app.js

Ensure to send requests to your routes to generate some traces:

curl http://localhost:8080/

curl http://localhost:8080/fibonacci/30

Refresh the Jaeger UI, and click the Service dropdown to select yourapplication:

Click Find traces to view the most recently collected traces for yourservice:

If you click on an individual trace you are presented with a breakdown of thespans contained in the trace:

This trace was generated for a request to /fibonacci/40, and it clearly showsthat approximately all the time generating a response was spent calculating thespecified Fibonacci number.

In a distributed scenario where all the downstream services were alsoinstrumented for tracing and pushing spans to the same Jaeger instance, you'llsee the entire request journey mapped out in Jaeger!

This demonstrates the general process of instrumenting an application withOpenTelemetry and seeing the timeline of how requests flow through your system,revealing the interactions between components.

Best practices for OpenTelemetry instrumentation

To ensure that OpenTelemetry's utility is maximized in your project, follow theguidelines below:

1. Avoid over-instrumentation

While auto-instrumentation offers convenience, exercise caution to avoidexcessive, irrelevant data that hampers troubleshooting. Selectively enableauto-instrumentation only for necessary libraries and be measured wheninstrumenting your code.

2. Instrument as you code

Embrace observability-driven development (ODD) by incorporating instrumentationwhile writing code. This ensures targeted instrumentation and prevents technicaldebt associated with retrofitting observability later.

3. Own your instrumentation

Application teams should take ownership of instrumenting their code. Theirintimate knowledge of the codebase ensures optimal instrumentation and effectivetroubleshooting.

4. Deploy an OpenTelemetry Collector

Utilize at least one Collector instance to centralize data collection andprocessing from various sources instead of sending telemetry data directly fromyour application. This streamlines data management, enables seamless backendswitching, and simplifies future observability adjustments through YAMLconfiguration updates.

Challenges of OpenTelemetry

Despite its immense potential and growing popularity, OpenTelemetry presentsseveral challenges that you need to consider before adopting it in yourorganization:

1. Maturity and stability

While the tracing component is fairly mature, many aspects of logs and metricssupport are still evolving. This can lead to inconsistencies, breaking changes,and a steeper learning curve for new adopters

2. Complexity

OpenTelemetry is a complex project with a wide range of features and components.The learning curve can be steep, particularly if you're new to observability ordistributed tracing concepts. Properly configuring and managing the Collectorcan also be challenging, requiring a deep understanding of its configurationoptions.

3. Instrumentation overhead

While automatic instrumentation simplifies the process, it can sometimesintroduce performance overhead, especially in high-traffic environments.Fine-tuning and optimizing instrumentation may be necessary to minimize theimpact on application performance.

4. Varying component quality

The evolving nature and varying quality of OpenTelemetry libraries anddocumentation pose a challenge, especially as new versions are releasedfrequently. The current inconsistency in maturity across components can lead tovarying user experiences depending on your specific needs and goals.

5. Documentation gaps

Documentation and best practices are still evolving, and there might be a lackof clear guidance for certain use cases or specific technologies. This can leadto trial and error, and slower adoption.

Final thoughts

I hope this article has helped you understand where OpenTelemetry fits in yourobservability strategy and how it provides a standardized vendor-neutral way tocollect telemetry signals from your application and infrastructure.

For even more information on OpenTelemetry, consider visitingtheir website, and digging deeper into theofficial documentation.

Thanks for reading!