Installing and Configuring the OpenTelemetry Collector
Installing and Configuring the OpenTelemetry Collector
The scope of the OpenTelemetry project encompasses how telemetry data is collected, processed, and transmitted. The OpenTelemetry project is not involved with how the data is stored, displayed, or used beyond the collection and transmission phases.
The OpenTelemetry Collector is an application written in Go. The GitHub readme does an excellent job of describing the collector:
The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process and export telemetry data. In addition, it removes the need to run, operate and maintain multiple agents/collectors in order to support open-source telemetry data formats (e.g. Jaeger, Prometheus, etc.) sending to multiple open-source or commercial back-ends.
Usable: Reasonable default configuration, supports popular protocols, runs and collects out of the box.
Performant: Highly stable and performant under varying loads and configurations.
Observable: An exemplar of an observable service.
Extensible: Customizable without touching the core code.
Unified: Single codebase, deployable as an agent or collector with support for traces, metrics and logs.
So the OpenTelemetry collector is a Go binary that does exactly what its name implies: it collects data and sends it to a back-end. In this post we’ll take a look at how it can be installed and configured.
The Collector can be deployed in two ways:
- As an Agent running on the same host as your instrumented application, either as a local service, sidecar, or daemonset.
- As a Gateway running in a dedicated server, container, or deployment.
It’s recommended to run as an Agent on every host within your environment, as the collector can attach additional information to the collected telemetry data.
If you have a large infrastructure and many hosts emitting OpenTelemetry data, you may want to add collector Gateways that accept the data from the local collector Agents. The gateways can then further process the data, such as applying tail based sampling, and it can simplify the egress management of the data.
Installing the Collector
There are a myriad of ways to install the collector. There are details for each method on this Getting Started page. There are options for installing via:
For our example, we’ll install on an Ubuntu server like so:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.53.0/otelcol_0.53.0_linux_arm64.deb sudo dpkg -i otelcol_0.53.0_linux_arm64.deb
Configuring the Collector
The default configuration file when installed on an Ubuntu server is located at
/etc/otelcol/config.yam and looks like this:
extensions: health_check: pprof: endpoint: 0.0.0.0:1777 zpages: endpoint: 0.0.0.0:55679 receivers: otlp: protocols: grpc: http: opencensus: # Collect own metrics prometheus: config: scrape_configs: - job_name: 'otel-collector' scrape_interval: 10s static_configs: - targets: ['0.0.0.0:8888'] jaeger: protocols: grpc: thrift_binary: thrift_compact: thrift_http: zipkin: processors: batch: exporters: logging: logLevel: debug service: pipelines: traces: receivers: [otlp, opencensus, jaeger, zipkin] processors: [batch] exporters: [logging] metrics: receivers: [otlp, opencensus, prometheus] processors: [batch] exporters: [logging] extensions: [health_check, pprof, zpages]
Note the top-level keys:
extensions receivers processors exporters service
Let’s talk about each of these sections individually.
Extensions are capabilities unrelated to the collection, processing, and exporting of telemetry data. We’re going to enable the local health check, pprof, and pages endpoints:
extensions: health_check: pprof: endpoint: 0.0.0.0:1777 zpages: endpoint: 0.0.0.0:55679
We will be able to reach the pprof page at
The pages endpoints are described in the pages extension README:
The collector exposes the following zPage routes: ServiceZ gives an overview of the collector services and quick access to the pipelinez, extensionz, and featurez zPages. The page also provides build and runtime information. PipelineZ brings insight on the running pipelines running in the collector. You can find information on type, if data is mutated and the receivers, processors and exporters that are used for each pipeline. ExtensionZ shows the extensions that are active in the collector. FeatureZ lists the feature gates available along with their current status and description. The TraceZ route is available to examine and bucketize spans by latency buckets for example (0us, 10us, 100us, 1ms, 10ms, 100ms, 1s, 10s, 1m] They also allow you to quickly examine error samples The Rpcz route is available to help examine statistics of remote procedure calls (RPCs) that are properly instrumented. For example when using gRPC
We can reach all of these pages via the local URLs:
http://localhost:55679/debug/servicez http://localhost:55679/debug/pipelinez http://localhost:55679/debug/extensionz http://localhost:55679/debug/featurez http://localhost:55679/debug/tracez http://localhost:55679/debug/rpcz
Receivers are how telemetry data gets into the collector. Some receivers can support multiple different metric types. For our example, we just want to accept data directly from our instrumented applications via OTLP, which is the OpenTelemetry Line Protocol format. We’ll enable both the gRPC and HTTP endpoints:
receivers: otlp: protocols: grpc: http:
Refer to the receiver README for more details. Currently there are 64 different receivers available in the opentelemetry-collector-contrib repo.
Note: configuring a receiver does not enable it. Receivers are enabled by adding the receiver name to a pipeline configuration.
Processors perform some action when telemetry data is received. There are processors for filtering, sampling, adding or editing attributes on telemetry data, and more. Some processors can support multiple different metric types. Check the opentelemetry-collector-contrib repo for available processors. We’re going to make sure sensitive information is scrubbed from trace spans in our example, and we’re going to configure the
batch processor as well:
processors: batch: resource: attributes: - key: auth-token action: delete
Note: configuring a processor does not enable it. Processors are enabled by adding the processor name to a pipeline configuration.
Exporters are how the telemetry data is sent to one or more backends/destinations. Some exporters can support multiple different metric types. In our example, we just want to log our traces to a file:
exporters: file: path: /tmp/otelcol_data.log logLevel: debug
Refer to the exporter README for more details. Currently there are 43 different exporters available in the opentelemetry-collector-contrib repo.
Note: configuring an exporter does not enable it. Exporters are enabled by adding the exporter name to a pipeline configuration.
The Service section defines what components are enabled from the extensions, receivers, processors, and exporters sections. There are three subsections of the
extensions sections is just a list of which extensions to enable. In our example, we’ll enable all that we configured earlier:
service: extensions: [health_check, pprof, zpages]
telemetry section is where metrics and logs of the collector itself can be configured. We’ll skip this for our example.
pipelines section is where we configure the
exporters for each telemetry type. The pipeline can be for the types:
logs. In our example, we’ll enable traces collection using the
otlp receiver, process using the
batch processor, and send it to the configured
service: extensions: [health_check, pprof, zpages] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [logging]
/etc/otelcol/config.yaml file now looks like this:
extensions: health_check: pprof: endpoint: 0.0.0.0:1777 zpages: endpoint: 0.0.0.0:55679 receivers: otlp: protocols: grpc: http: processors: batch: resource: attributes: - key: auth-token action: delete exporters: file: path: /tmp/otelcol_data.log logLevel: debug service: extensions: [health_check, pprof, zpages] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [logging]
Restart the otelcol service:
sudo service otelcol restart.
Now we’ll see the trace payloads from our applications instrumented with OpenTelemetry in
/tmp/otelcol_data.log. Check out our posts on Configuring OpenTelemetry in Python and Configuring OpenTelemetry in Ruby if you need help instrumenting your Ruby or Python apps.
If you’re interested in learning more about OpenTelemetry, Observability and what we’re working on at Scout, sign up for our Observability newsletter at scoutapm.com/observability.