Features

Scout APM

Scout is Application Monitoring built for modern development teams. It’s built to provide the fastest path to a slow line-of-code. Signup for a trial.

App Performance Overview

The overview page provides an at-a-glance, auto-refreshing view of your app’s performance and resource usage (mean response time by category, 95th percentile response time, throughput, error rate, and more). You can quickly dive into endpoint activity via click-and-drag (or pinch-and-expand with a mobile device) on the overview chart.

overview

Additionally, you can compare metrics in the overview chart and see how your app’s performance compares to different time periods.

Endpoint Details

You can view metrics for specific controller-action and background job workers. There is a similar chart interaction to the App Performance Overview page, with one difference: your selection will render an updated list of transaction traces that correspond to the selected time period: stream

You can sort traces by response time, object allocations, date, and more.

Transaction Traces

Scout collects detailed transactions across your web endpoints and background jobs automatically. The transaction traces provide a number of visual queues to direct you to hotspots. Dig into bottlenecks - down to the line-of-code, author, commit date, and deploy time - from this view.

transaction traces

SQL Queries

Scout captures a sanitized version of SQL queries. Click the “SQL” button next to a call to view details.

stream show sql

Don’t see an SQL button next to a database query?

Scout collects a sanitized version of SQL queries and displays these in transaction traces. To limit agent overhead sanitizing queries, we do not collect query statements with more than 16k characters.

Code Backtraces

You’ll see “CODE” buttons next to method calls that are >= 500 ms. If you’ve enabled the GitHub integration, you can see the line-of-code, associated SQL or HTTP endpoint (if applicable), author, commit date, and deploy time for the relevant slow code.

stream show git

If you don’t enable the GitHub integration, you’ll see a backtrace.

Trace Views

There are two displays for showing the details of a transaction trace:

Summary View

Method calls are aggregated together and listed from most expensive to least expensive. The time displayed is the total time across all calls (not the time per-call).

stream show breakdown

Timeline View

See the execution order of your code.

trace timeline

The timeline view is especially helpful for:

Upgrading to the timeline view

For older Ruby agents, if you see a message in the UI prompting you to upgrade, follow our Ruby agent upgrade instructions to update to the latest agent, which supports sending the timeline trace format.

Timeline view limitations

Trace Explorer

What was the slowest request yesterday? How has the app performed for user@domain.com? Which endpoints are generating the bulk of slow requests? Trace Explorer lets you quickly filter the transaction traces collected by Scout, giving you answers to your unique questions.

crossfilter

Trace Explorer is accessed via the “Traces” navigation link when viewing an app.

How to use Trace Explorer

There are two main areas of Trace Explorer:

Memory Bloat Detection

If a user triggers a request to your Rails application that results in a large number of object allocations (example: loading a large number of ActiveRecord objects), your app may require additional memory. The additional memory required to load the objects in memory is released back very slowly. Therefore, a single memory-hungry request will have a long-term impact on your Rails app’s memory usage.

There are 3 specific features of Scout to aid in fixing memory bloat.

Memory Bloat Insights

The Insights area of the dashboard identifies controller-actions and background jobs that have triggered significant memory increases. An overview of the object allocation breakdown by tier (ActiveRecord, ActionView, etc) is displayed on the dashboard.

memory insights

Memory Traces

When inspecting a transaction trace, you’ll see a “Memory Allocation Breakdown” section:

memory trace

For perspective, we display how this trace’s allocations compare to the norm.

Alerting

alerts_chart

Alerting keeps your team updated if your app’s performance degrades. Alerts can be configured on the app as a whole as well as on individual endpoints and background jobs. Metrics include:

Web Requests:

Background Jobs:

Alert conditions

Configure alert conditions via the “Alerts” pill in the UI:

alerts_chart

Notification groups

Alerts are sent to a notification group, which is composed of notification channels. You can configure these under your org’s settings menu:

alert notification groups

Deploy Tracking

deploy tracking

Correlate deploys with your app’s performance: Scout’s GitHub-enhanced deploy tracking makes it easy to identify the Git branch or tag running now and which team members contributed to every deploy.

Sorting

You can sort by memory allocations throughout the UI: from the list of endpoints, to our pulldowns, to transaction traces.

memory sort

Request Queuing

Our agents can measure the time it takes a request to reach your application from farther upstream (a load balancer or web server). This appears in Scout as “Request Queuing” and provides an indication of your application’s capacity. Large request queuing time is an indication that your app needs more capacity.

request queuing

To see this metric within Scout, you need to configure your upstream software, adding an HTTP header that our agent reads. This is typically a one-line change.

Setup

HTTP Header

The Scout agent depends on an HTTP request header set by an upstream load balancer (ex: HAProxy) or web server (ex: Apache, Ngnix).

Protip: We suggest adding the header as early as possible in your infrastructure. This ensures you won’t miss performance issues that appear before the header is set.

The agent will read any of the following headers as the start time of the request:

X-Queue-Start, X-Request-Start, X-QUEUE-START, X-REQUEST-START, x-queue-start, x-request-start

Include a value in the format t=MICROSECONDS_SINCE_EPOCH where MICROSECONDS_SINCE_EPOCH is an integer value of the number of microseconds that have elapsed since the beginning of Unix epoch.

Nearly any front-end HTTP server or load balancer can be configured to add this header. Some examples are below.

Heroku

Time in queue is automatically collected for apps deployed on Heroku. This measures the time from when a request hits the Heroku router and when your app begins processing the request.

Apache

Apache’s mod_headers module includes a %t variable that is formatted for Scout usage. To enable request queue reporting, add this code to your Apache config:

RequestHeader set X-Request-Start "%t"
Apache Request Queuing and File Uploads

If you are using Apache, you may observe a spike in queue time within Scout for actions that process large file uploads. Apache adds the X-Request-Start header as soon as the request hits Apache. So, all of the time spent uploading a file will be reported as queue time.

This is different from Nginx, which will first buffer the file to a tmp file on disk, then once the upload is complete, add headers to the request.

HAProxy

HAProxy 1.5+ supports timestamped headers and can be set in the frontend or backend section. We suggest putting this in the frontend to get a more accurate number:

http-request set-header X-Request-Start t=%Ts

Nginx

Nginx 1.2.6+ supports the use of the #{msec} variable. This makes adding the request queuing header straightforward.

General Nginx usage:

proxy_set_header X-Request-Start "t=${msec}";

Passenger 5+:

passenger_set_header X-Request-Start "t=${msec}";

Older Passsenger versions:

passenger_set_cgi_param X-REQUEST-START "t=${msec}";

Note: The Nginx option is local to the location block, and isn’t inherited.

Context

Context lets you see the forest from the trees. For example, you can add custom context to answer critical questions like:

Adding custom context is easy - learn how via Ruby, Elixir, Python, PHP, and Node

Context information is displayed in two areas:

Endpoints Performance

Endpoints Overview

The endpoints area within Scout provides a sortable view of your app’s overall performance aggregated by endpoint name. Click on an endpoint to view details.

endpoints overview

Time Comparisons

You can easily compare the performance of your application between different time periods via the time selection on the top right corner of the UI.

time compare

Database Monitoring

Supported:

Ruby Python PHP Elixir

When the database monitoring feature is enabled, you’ll gain access to both a high-level overview of your database query performance and detailed information on specific queries. Together, these pieces make it easier to get to the source of slow query performance.

Database Queries Overview

The high-level view helps you identify where to start:

overview

The chart at the top shows your app’s most time-consuming queries over time. Beneath the chart, you’ll find a sortable list of queries grouped by a label (for Rails apps, this is the ActiveRecord model and operation) and the caller (a web endpoint or a background job):

This high-level view is engineered to reduce the investigation time required to:

Zooming

If there is a spike in time consumed or throughput, you can easily see what changed during that period. Click and drag over the area of interest on the chart:

zoom

Annotations are added to the queries list when zooming:

Database Events

Scout highlights significant events in database performance in the sidebar. For example, if time spent in database queries increases dramatically, you’ll find an insight here. Clicking on an insight jumps to the time window referenced by the insight.

Database Query Details

After identifying an expensive query, you need to see where the query is called and the underlying SQL. Click on a query to reveal details:

detail

You’ll see the raw SQL and a list of individual query execution times that appeared in transaction traces. Scout collects backtraces on queries consuming more than 500 ms. If we’ve collected a backtrace for the query, you’ll see an icon next to the timing information. Click on one of the traces to reveal that trace in a new window:

trace

The source of that trace is immediately displayed.

Slow Query Insights

When the database monitoring feature is enabled (with the pro plan), a new “Slow Query” insight is activated on your app dashboard:

slow query insights

This insight analyzes your queries in three dimensions, helping you focus on database optimizations that will most improve your app:

  1. Which queries are most impacting app performance? This is based on the total time consumed of each query, where time consumed is the average query latency multiplied by the query throughput.
  2. Which queries are significant bottlenecks inside web endpoints and background jobs? A single query that is responsible for a large percentage of the time spent in a transaction is a great place to investigate for a performance win.
  3. Which queries are consistently slow? These are queries that have a high average latency.

Pricing

For direct users, Database Monitoring is available as pro plan feature. For Heroku users, it’s available as an addon. See your billing page for pricing information.

Database Monitoring Installation

Update - or install - the scout_apm gem in your application. There’s no special libraries to install on your database servers.

Database Monitoring Library Support

Scout currently monitors queries executed via ActiveRecord, which includes most relational databases (PostgreSQL, MySQL, etc).

What does SQL#other mean?

Some queries may be identified by a generic SQL#other label. This indicates our agent was unable to generate a friendly label from the raw SQL query. Ensure you are running version 2.3.3 of the scout_apm gem or higher as this release includes more advanced query labeling logic.

Digest Email

At a frequency of your choice (daily or weekly), Scout crunches the numbers on your app’s performance (both web endpoints and background jobs). Performance is compared to the previous week, and highlights are mentioned in the email.

digest

The email identifies performance trends, slow outliers, and attempts to narrow down issues to a specific cause (like slow HTTP requests to another service).

Chart Embeds

You can embed an app’s overview chart inside another web page (ex: an internal key metrics dashboard):

  1. Access the application dashboard within the Scout UI.
  2. Adjust the timeframe and metrics to those you’d like to include in the embedded chart.
  3. Click the embed icon and copy the relevant code.
chart_embed

Note that you’ll need to update the provided iframe url with a Scout API key.

When clicking on an embedded chart, you’ll be redirected to the relevant application.