START FOR FREE
LOGIN

Exploring OpenTelemetry Configuration in Ruby

by | performance

OpenTelemetry is enabling a revolution in how Observability data is collected and transmitted. See our What Is OpenTelemetry post on why this is an important inflection point in the Observability space.

In this post, we’ll walk through how to configure the OpenTelemetry Gems within a Rails app.

Securing Your SDK Installation

The first step is to install the OpenTelemetry Gems:

Gemfile.rb:

gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'
gem 'opentelemetry-instrumentation-all'

config/initializers/opentelemetry.rb:

require 'opentelemetry/sdk'
require 'opentelemetry/instrumentation/all'

The sdk provides the interface to which you’ll interact with the OpenTelemetry instrumentation and exporter Gems. The GitHub repo for the Ruby SDK can be found here

The instrumentation Gem provides the code that measures functions of popular Gems you are using within your own application. The GitHub repo for the Ruby OpenTelemetry instrumentation can be found here. There are out-of-the-box instruments for about 30 libraries as of January 2022.

The absolute minimum configuration required to just see the raw traces generated from the instrumentation traces printed to the console locally is

config/initializers/opentelemetry.rb:

ENV['OTEL_TRACES_EXPORTER'] = 'console'
OpenTelemetry::SDK.configure do |c|
  c.use_all
end

This will apply all available instrumentation libraries and will skip any libraries which you don’t use in your application. When you start up your Rails app and send a request through, you’ll see the Pretty-Printed Traces and Spans recorded from OpenTelemetry in the console.

Of course, this isn’t very practical, other than verifying the instrumentation and tracing is working. You’ll also notice that there are quite a few log messages about failing instrumentation:

*** Disabling caching for development environment
=> Booting Puma
=> Rails 6.0.4.4 application starting in development
=> Run `rails server --help` for more startup options
I, [2022-02-01T10:37:28.663031 #1170243]  INFO -- : Instrumentation: OpenTelemetry::Instrumentation::Rack was successfully installed
I, [2022-02-01T10:37:28.677835 #1170243]  INFO -- : Instrumentation: OpenTelemetry::Instrumentation::ActionPack was successfully installed
I, [2022-02-01T10:37:28.678653 #1170243]  INFO -- : Instrumentation: OpenTelemetry::Instrumentation::ActionView was successfully installed
I, [2022-02-01T10:37:28.679446 #1170243]  INFO -- : Instrumentation: OpenTelemetry::Instrumentation::ActiveJob was successfully installed
I, [2022-02-01T10:37:28.685357 #1170243]  INFO -- : Instrumentation: OpenTelemetry::Instrumentation::ActiveRecord was successfully installed
W, [2022-02-01T10:37:28.685425 #1170243]  WARN -- : Instrumentation: OpenTelemetry::Instrumentation::Bunny failed to install
W, [2022-02-01T10:37:28.685469 #1170243]  WARN -- : Instrumentation: OpenTelemetry::Instrumentation::LMDB failed to install
W, [2022-02-01T10:37:28.685489 #1170243]  WARN -- : Instrumentation: OpenTelemetry::Instrumentation::HTTP failed to install
W, [2022-02-01T10:37:28.685504 #1170243]  WARN -- : Instrumentation: OpenTelemetry::Instrumentation::Koala failed to install
...<snip>...

Fine-Tuning Instrumentation Configuration

Let’s clean up the instrumentation configuration and explicitly list the instruments we want applied. We’ll also pass in a configuration option to the Postgres instrumentation Gem:

config/initializers/opentelemetry.rb:

OpenTelemetry::SDK.configure do |c|
  ##### Instruments
  c.use 'OpenTelemetry::Instrumentation::Rack'
  c.use 'OpenTelemetry::Instrumentation::ActionPack'
  c.use 'OpenTelemetry::Instrumentation::ActionView'
  c.use 'OpenTelemetry::Instrumentation::ActiveJob'
  c.use 'OpenTelemetry::Instrumentation::ActiveRecord'
  c.use 'OpenTelemetry::Instrumentation::ConcurrentRuby'
  c.use 'OpenTelemetry::Instrumentation::Faraday'
  c.use 'OpenTelemetry::Instrumentation::HttpClient'
  c.use 'OpenTelemetry::Instrumentation::Net::HTTP'
  c.use 'OpenTelemetry::Instrumentation::PG', {
    # By default, this instrumentation includes the executed SQL as the `db.statement`
    # semantic attribute. Optionally, you may disable the inclusion of this attribute entirely by
    # setting this option to :omit or sanitize the attribute by setting to :obfuscate
    db_statement: :obfuscate,
  }
  c.use 'OpenTelemetry::Instrumentation::Rails'
  c.use 'OpenTelemetry::Instrumentation::Redis'
  c.use 'OpenTelemetry::Instrumentation::RestClient'
  c.use 'OpenTelemetry::Instrumentation::RubyKafka'
  c.use 'OpenTelemetry::Instrumentation::Sidekiq'
end

Notice that we set db_statement: :obfuscate within a block of the c.use method after specifying the class name OpenTelemetry::Instrumentation::PG. Some instrumentation libraries have configuration options like this. Check the appropriate Gem Readme for options. In this case, we are sanitizing the database query statements the instrument records and reports in the OpenTelemetry payload:

"db.statement"=>
    "SELECT 1 AS one FROM "widget" WHERE "widget"."thingy_id" = $1 AND "widget"."kind" = $2 AND "widget"."deleted_at" IS NULL LIMIT $3"

Resource Configuration: An Overview

From the OpenTelemetry spec, a Resource:

Resource captures information about the entity for which telemetry is recorded. For example, metrics exposed by a Kubernetes container can be linked to a resource that specifies the cluster, namespace, pod, and container name.

Resource may capture an entire hierarchy of entity identification. It may describe the host in the cloud and specific container or an application running in the process.

In other words, the resource attributes attached to a trace provide the metadata to identify where the trace was generated, and any other useful context that enables the user searching traces to filter and drill down to specific values of the attributes. E.g. Show me traces:

  • Where cloud = AWS
  • And region = us-west-1
  • And instance_id = i-12334567
  • And service_name = rails-production
  • And version = 5.9.8

config/initializers/opentelemetry.rb:

OpenTelemetry::SDK.configure do |c|
  ##### Resource configuration
  # These can also be set via ENV var. E.g.
  # env OTEL_RESOURCE_ATTRIBUTES="foo=bar,baz=quux"
  #
  # Some, like SERVICE_NAME are common enough to have their own shortcut setters
  # env OTEL_SERVICE_NAME=rails
  # c.service_name = 'rails' # Shortcut setter alternative below
  c.resource = OpenTelemetry::SDK::Resources::Resource.create({
    OpenTelemetry::SemanticConventions::Resource::SERVICE_NAMESPACE => 'scout_apm',
    OpenTelemetry::SemanticConventions::Resource::SERVICE_NAME => 'rails',
    OpenTelemetry::SemanticConventions::Resource::SERVICE_INSTANCE_ID => Socket.gethostname,
    OpenTelemetry::SemanticConventions::Resource::SERVICE_VERSION => "5.9.8",
    "scout.id" => "my_scout_auth_id",
    "scout.key" => "my_scout_auth_key",
  })

  ##### Instruments
...<snip>...
end

In OpenTelemetry, Semantic Conventions are resource attributes that are formalized within the OpenTelemetry specification. Some are required to be set by the SDKs and some are optional, but the attributes defined as Semantic Conventions means that attribute key or value is so commonly used within OpenTelemetry that is has been formally written into the Specification so that it can be consistently applied throughout any system using OpenTelemetry.

In the example above, you can see that there are multiple ways to set values for the Semantic Conventions in your Rails app.

ENVIRONMENT VARIABLES

For some of the Semantic Convention attributes, there are special environment variables you can set to define the values. E.g. OTEL_SERVICE_NAME=rails-production

There is also a catch-all environment variable that lets you set multiple resource attributes: OTEL_RESOURCE_ATTRIBUTES="service_name=rails-production,service_namespace=scout_apm

IN CODE

Within the OpenTelemetry::SDK.configure block:

config/initializers/opentelemetry.rb:

OpenTelemetry::SDK.configure do |c|
  c.resource = OpenTelemetry::SDK::Resources::Resource.create(
    OpenTelemetry::SemanticConventions::Resource::SERVICE_NAME => 'rails-production',
    "scout.id" => "my_scout_auth_id"
end

The OpenTelemetry::SemanticConventions::Resource::SERVICE_NAME constant simply resolves to service_name. You can specify any resource attribute keys using a string, like we did with scout.id.

Processors and Exporters

When traces are finished, we need to do something with them. This is where Processors and Exporters come in.

config/initializers/opentelemetry.rb:

require 'opentelemetry/exporter/otlp'

# The OTLP endpoint is specified by ENV var on boot.
# Local development: env OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:55681/v1/traces bundle exec ...

##### Exporter and Processor configuration
otel_exporter = OpenTelemetry::Exporter::OTLP::Exporter.new
processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(otel_exporter)

OpenTelemetry::SDK.configure do |c|
  ##### Exporter and Processor configuration
  c.add_span_processor(processor)

  ##### Resource configuration
...<snip>...
end

Notice that we added a new require for the opentelemetry/exporter/otlp library.

Processors are responsible for handling the traces after the trace is complete. By default, the Ruby OpenTelemetry SDK uses a SimpleSpanProcessor which takes each trace and sends it to the exporter one-by-one.

You’ll likely want to use the BatchSpanProcessor, which batches up the processing of spans and results in more efficient exporting of those traces.

Exporters are responsible for sending the traces, or batches of traces, to another destination. OpenTelemetry has its own transmission protocol for sending data, called OTLP. In the example above, we use the OTLP protocol to send the trace batches to a collector or SaaS service like Scout. There are also exporters available for Zipkin and Jaeger.

To specify the OTLP endpoint, you will use an environment variable. If you have the OptenTelemetry Collector running locally on port 55681, you would set the variable like so: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:55681/v1/traces. For sending to a SaaS service directly via OTLP, just replace the URL in the local collector example with the URL provided by your provider.

The Last Step: Final Configuration

The complete configuration should now look like this:

require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
require 'opentelemetry/instrumentation/all'

##### Exporter and Processor configuration
otel_exporter = OpenTelemetry::Exporter::OTLP::Exporter.new
processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(otel_exporter)

OpenTelemetry::SDK.configure do |c|
  ##### Exporter and Processor configuration
  c.add_span_processor(processor) # Created above this SDK.configure block

  ##### Resource configuration
  c.resource = OpenTelemetry::SDK::Resources::Resource.create({
    OpenTelemetry::SemanticConventions::Resource::SERVICE_NAMESPACE => 'scout_apm',
    OpenTelemetry::SemanticConventions::Resource::SERVICE_NAME => 'rails',
    OpenTelemetry::SemanticConventions::Resource::SERVICE_INSTANCE_ID => Socket.gethostname,
    OpenTelemetry::SemanticConventions::Resource::SERVICE_VERSION => "0.0.0",
    "scout.id" => "my_scout_auth_id",
    "scout.key" => "my_scout_auth_key",
  })

  ##### Instruments
  c.use 'OpenTelemetry::Instrumentation::Rack'
  c.use 'OpenTelemetry::Instrumentation::ActionPack'
  c.use 'OpenTelemetry::Instrumentation::ActionView'
  c.use 'OpenTelemetry::Instrumentation::ActiveJob'
  c.use 'OpenTelemetry::Instrumentation::ActiveRecord'
  c.use 'OpenTelemetry::Instrumentation::ConcurrentRuby'
  c.use 'OpenTelemetry::Instrumentation::Faraday'
  c.use 'OpenTelemetry::Instrumentation::HttpClient'
  c.use 'OpenTelemetry::Instrumentation::Net::HTTP'
  c.use 'OpenTelemetry::Instrumentation::PG', {
    db_statement: :obfuscate,
  }
  c.use 'OpenTelemetry::Instrumentation::Rails'
  c.use 'OpenTelemetry::Instrumentation::Redis'
  c.use 'OpenTelemetry::Instrumentation::RestClient'
  c.use 'OpenTelemetry::Instrumentation::RubyKafka'
  c.use 'OpenTelemetry::Instrumentation::Sidekiq'
end

You can now boot up your rails app locally, specifying the OTLP endpoint for traces to be sent:

env OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:55681/v1/traces bundle exec rails s

In the next post, we’ll cover setting up the Collector on your local machine to see these traces sent via OTLP.

Distributed Tracing and the Collector

In this post, we hooked up OpenTelemetry for the back-end Rails code. In our next post, we cover hooking up OpenTelemetry to the front-end, enabling Distributed Tracing, and show you how to send these to an OpenTelemetry Collector.