Logs [Beta]

Installation

Our logging solution aims to “just work” with Rails applications. Other frameworks and languages coming soon!

1. Add the scout_apm_ruby_logging gem to your gemfile.

2. Configure a few environment variables:

    SCOUT_LOGS_MONITOR=true
    SCOUT_LOGS_INGEST_KEY=aaaa-1111-aaaa-1111 # Provided in Scout settings
   

3. Deploy!

Further configuration options are available, but the above is the minimum required to get started.

Overview

Scout’s logging is designed with ease-of-use as a top priority. Sending logs to us and viewing them in the UI should be as simple as possible. By working with our existing agents, we can also gather more context and automatically apply it to your application logs. This gives you extra power to filter and search through your logs, and to correlate them with other performance data. Some highlights:

• Filter logs by entrypoint (i.e. Controller originating the activity)
• Filter logs by Custom Context (attributes - anything you have added to the Scout APM agent)
• Fast in-memory exploration

Functionality

Visiting the Logs page will begin loading the log records that we have received according to the timeframe you have selected, e.g. “Past 3 hours.” Logs will load from most-recent to oldest. We load 10K records initially, more can be loaded as needed. As the logs load, you can rapidly filter to desired Severity Levels and search/filter via regex applied to the message body and attributes. The time window can also be narrowed via horizontal selection (“brush”) on the chart. These filters apply to logs that are already loaded into the browser.

When you determine interesting lines, you can also use the pre-load filters to screen logs closer to the log storage layer, allowing you to scan longer timeframes without pulling too many records into the browser. This two-phase approach can be a powerful and flexible way to efficiently seek through a very large corpus of log records. We hope you love it!

Filtering

Pre-Load Filters

The Logs Filtering allows you to limit the logs that are loaded into the Logs View. These filters are applied to the logs in our storage system before loading them into the application. Filtering by time can especially reduce the amount of data processed and returned by our application. Keep in mind that rapid filtering and manipulation can be further performed within the Logs Table after the Logs have been loaded.

Logs can be filtered by:

• Date and Time: Using the Scout time selector, as usual.
• Message Content: A simple CONTAINS filter is available.
• Attributes: Values of any attributes that are present in the logs can be used to filter.

For Attribute filters, you will need to select the attribute key and then provide a value to filter by. However, we only display a couple of predetermined attributes, with the use of custom context, more attributes can be added to the logs. You can specify additional filters for custom log attributes and values. See the below gif for adding the ‘org_id’ custom context we have added to our application as a filter.

The four default attributes:

• ID: Log record id
• Body: Log message / body
• Severity: Log severity – debug, info, etc.
• ServiceName: The service name, usually the process name – unicorn worker, sidekiq worker, etc.

After changing any of these filters, hitting “Load” will discard the current in-memory logs and begin loading new data matching the filter criteria.

Default Attributes

The Scout Logs agent enriches Rails logs with a few attributes by default:

• entrypoint: The top-level action (e.g. Controller class, in Rails) that the log was generated from.
• location: The file and line number where the log was generated.

In addition, it will capture all key-value pairs from any Custom Context that you have set. This means logs can be filtered by any Custom Context attributes.

Logs List

As records are loading, the Logs List allows you to filter your log data in straightforward ways. These filters also determine the data sent to the rest of the statistics and charts on the screen, allowing you to focus on the most relevant Logs for any given visualization.

Severity Filter

Select any combination of severity levels to filter the logs.

Regex Filter

The Regex Filter allows you to filter the Logs List using regex that evaluates against message content as well as log Attributes (both keys and values). We hope it lets you quickly narrow things down, whatever you might want to narrow by.

Brush Tool

On the chart, you can select the brush tool to click and drag across a time range to filter the List.

Loading Controls

At the top of the Logs List are a few controls that allow you to manage the loading of data into the Logs List. These controls are useful for managing the amount of data that is loaded into the Logs List at any given time.

• Progress Info: This section indicates the amount of data that has been returned vs scanned.
• Loading Bar: This bar indicates the progress of the data being loaded. You can hover over to get an approximate numerical value.
• Pause/Resume: These buttons allow you to pause and resume the loading of data. This is useful if you want to stop the loading of data to inspect the data that has already been loaded or if you don’t want to load the default 10K Logs per load.
• Full Screen: This button allows you to expand the Logs List to full screen mode.

Usage & Billing

Usage can be broken out into two categories. Write usage and read usage. Write usage is the total bytes of uncompressed data sent to our servers in OTLP format. Read usage is the number of bytes of total uncompressed data that we traversed when performing an operation.

To view your usage, click the cog in the bottom left, which will display a list of options to choose from. Within this list click “View Logs Usage”. Here, you will have 14 days of both write and read usage streamed in.

Common Configurations

The following configuration settings are available. These can be set in the scout_apm.yml configuration file or as environment variables with the SCOUT_ prefix, e.g. SCOUT_LOGS_INGEST_KEY.

Only logs_ingest_key is required. The rest are optional.

Additional Configurations

These settings are internal and are not typically needed for normal operation, but if things like filesystem access cause issues, the following may be useful.

Installation: The Manual Way

Scout’s logging solution is based on OpenTelemetry and the OTLP protocol. If you are running an Otel Collector, we can be an endpoint of a logs pipeline. You can theoretically send us log records from any type of service and we will group them according to the provided Ingest Key within Scout’s logging UI. If using the collector directly, you can add your scout_logs_ingest_key to the headers of the OTLP exporter as x-scout-key, or as an attribute in code where you configure OpenTelemetry as scout.key.

# otel-collector-config.yaml
receivers:
  ...
  filelog:
    include: [ /var/log/*.log ]

...

exporters:
  otlp:
    endpoint: https://otlp.scoutotel.com:4317
    headers:
      x-scout-key: aaaa-1111-aaaa-1111 # Provided in Scout settings

...

service:
  pipelines:
    ...
    logs:
      receivers: [otlp, filelog]
      processors: []
      exporters: [otlp]

FAQ

How does this work?

Our logging agent will download and configure the Otel Collector and run it on your server. The Collector will then collect logs from your application (either specified as explicit files or automatically created by our agent) and send them to Scout. The logs will be available in the Scout UI for you to search and filter. For Rails, we will automatically set a custom log formatter to include additional context in the logs sent to us. Your original logs will not be altered.

The Ruby logging agent is open source and available on GitHub.

What is the retention period on this log data?

Scout retains your log data for 14 days. If you require longer-term storage, please let us know as we would like to create options for our customers, but at this point you will need to also send them to an alternative location.