Installing and Configuring the OpenTelemetry Collector

Dave Anderson  on June 13, 2022

Overview

The scope of the OpenTelemetry project encompasses how telemetry data is collected, processed, and transmitted. The OpenTelemetry project is not involved with how the data is stored, displayed, or used beyond the collection and transmission phases.

The OpenTelemetry Collector is an application written in Go. The GitHub readme does an excellent job of describing the collector:

The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process and export telemetry data. In addition, it removes the need to run, operate and maintain multiple agents/collectors in order to support open-source telemetry data formats (e.g. Jaeger, Prometheus, etc.) sending to multiple open-source or commercial back-ends.

Objectives:

Usable: Reasonable default configuration, supports popular protocols, runs and collects out of the box.

Performant: Highly stable and performant under varying loads and configurations.

Observable: An exemplar of an observable service.

Extensible: Customizable without touching the core code.

Unified: Single codebase, deployable as an agent or collector with support for traces, metrics and logs.

So the OpenTelemetry collector is a Go binary that does exactly what its name implies: it collects data and sends it to a back-end. In this post we’ll take a look at how it can be installed and configured.

Deployment Types

The Collector can be deployed in two ways:

  1. As an Agent running on the same host as your instrumented application, either as a local service, sidecar, or daemonset.
  2. As a Gateway running in a dedicated server, container, or deployment.

It’s recommended to run as an Agent on every host within your environment, as the collector can attach additional information to the collected telemetry data.

If you have a large infrastructure and many hosts emitting OpenTelemetry data, you may want to add collector Gateways that accept the data from the local collector Agents. The gateways can then further process the data, such as applying tail based sampling, and it can simplify the egress management of the data.

Installing the Collector

There are a myriad of ways to install the collector. There are details for each method on this Getting Started page. There are options for installing via:

For our example, we’ll install on an Ubuntu server like so:

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.53.0/otelcol_0.53.0_linux_arm64.deb
sudo dpkg -i otelcol_0.53.0_linux_arm64.deb

Configuring the Collector

The default configuration file when installed on an Ubuntu server is located at /etc/otelcol/config.yam and looks like this:

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

  opencensus:

  # Collect own metrics
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otel-collector'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

  jaeger:
    protocols:
      grpc:
      thrift_binary:
      thrift_compact:
      thrift_http:

  zipkin:

processors:
  batch:

exporters:
  logging:
    logLevel: debug

service:

  pipelines:

    traces:
      receivers: [otlp, opencensus, jaeger, zipkin]
      processors: [batch]
      exporters: [logging]

    metrics:
      receivers: [otlp, opencensus, prometheus]
      processors: [batch]
      exporters: [logging]

  extensions: [health_check, pprof, zpages]

Note the top-level keys:

extensions
receivers
processors
exporters
service

Let’s talk about each of these sections individually.

EXTENSIONS

Extensions are capabilities unrelated to the collection, processing, and exporting of telemetry data. We’re going to enable the local health check, pprof, and pages endpoints:

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

We will be able to reach the pprof page at http://localhost:1777/debug/pprof:

The pages endpoints are described in the pages extension README:

The collector exposes the following zPage routes:

ServiceZ gives an overview of the collector services and quick access to the pipelinez, extensionz, and featurez zPages. The page also provides build and runtime information.

PipelineZ brings insight on the running pipelines running in the collector. You can find information on type, if data is mutated and the receivers, processors and exporters that are used for each pipeline.

ExtensionZ shows the extensions that are active in the collector.

FeatureZ lists the feature gates available along with their current status and description.

The TraceZ route is available to examine and bucketize spans by latency buckets for example

(0us, 10us, 100us, 1ms, 10ms, 100ms, 1s, 10s, 1m] They also allow you to quickly examine error samples

The Rpcz route is available to help examine statistics of remote procedure calls (RPCs) that are properly instrumented. For example when using gRPC

We can reach all of these pages via the local URLs:

http://localhost:55679/debug/servicez
http://localhost:55679/debug/pipelinez
http://localhost:55679/debug/extensionz
http://localhost:55679/debug/featurez
http://localhost:55679/debug/tracez
http://localhost:55679/debug/rpcz

RECEIVERS

Receivers are how telemetry data gets into the collector. Some receivers can support multiple different metric types. For our example, we just want to accept data directly from our instrumented applications via OTLP, which is the OpenTelemetry Line Protocol format. We’ll enable both the gRPC and HTTP endpoints:

receivers:
  otlp:
    protocols:
      grpc:
      http:

Refer to the receiver README for more details. Currently there are 64 different receivers available in the opentelemetry-collector-contrib repo.

Note: configuring a receiver does not enable it. Receivers are enabled by adding the receiver name to a pipeline configuration.

PROCESSORS

Processors perform some action when telemetry data is received. There are processors for filtering, sampling, adding or editing attributes on telemetry data, and more. Some processors can support multiple different metric types. Check the opentelemetry-collector-contrib repo for available processors. We’re going to make sure sensitive information is scrubbed from trace spans in our example, and we’re going to configure the batch processor as well:

processors:
  batch:
  resource:
    attributes:
    - key: auth-token
      action: delete

Note: configuring a processor does not enable it. Processors are enabled by adding the processor name to a pipeline configuration.

EXPORTERS

Exporters are how the telemetry data is sent to one or more backends/destinations. Some exporters can support multiple different metric types. In our example, we just want to log our traces to a file:

exporters:
  file:
    path: /tmp/otelcol_data.log
    logLevel: debug

Refer to the exporter README for more details. Currently there are 43 different exporters available in the opentelemetry-collector-contrib repo.

Note: configuring an exporter does not enable it. Exporters are enabled by adding the exporter name to a pipeline configuration.

SERVICE

The Service section defines what components are enabled from the extensions, receivers, processors, and exporters sections. There are three subsections of the service section:

The extensions sections is just a list of which extensions to enable. In our example, we’ll enable all that we configured earlier:

service:
      extensions: [health_check, pprof, zpages]

The telemetry section is where metrics and logs of the collector itself can be configured. We’ll skip this for our example.

The pipelines section is where we configure the receivers, processors, and exporters for each telemetry type. The pipeline can be for the types: metrics, traces, or logs. In our example, we’ll enable traces collection using the otlp receiver, process using the batch processor, and send it to the configured logging exporter:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Wrapping Up

Our /etc/otelcol/config.yaml file now looks like this:

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  resource:
    attributes:
    - key: auth-token
      action: delete

exporters:
  file:
    path: /tmp/otelcol_data.log
    logLevel: debug

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Restart the otelcol service: sudo service otelcol restart.

Now we’ll see the trace payloads from our applications instrumented with OpenTelemetry in /tmp/otelcol_data.log. Check out our posts on Configuring OpenTelemetry in Python and Configuring OpenTelemetry in Ruby if you need help instrumenting your Ruby or Python apps.

If you’re interested in learning more about OpenTelemetry, Observability and what we’re working on at Scout, sign up for our Observability newsletter at scoutapm.com/observability.