Anomaly Monitoring

Early beta — Anomaly Monitoring is in early beta and rolling out now. Want access for your org? Email support@scoutapm.com and we’ll get you set up.

Anomaly Monitoring watches your selected endpoints and background jobs and tells you when they start behaving unusually — even if you never set a threshold. Instead of asking “is response time above 500ms?" it asks “is response time meaningfully different from the last 24 hours, accounting for this same time last week?"

Use it alongside traditional alerting for endpoints where you don’t know what “normal” looks like, or where what’s normal changes over time.

How it works

Anomaly Monitoring is opt-in per endpoint and per background job, with a per-organization cap so you can focus it on what matters.

For each monitored endpoint or job, Scout looks at the average and 95th percentile response times.

Every minute, Scout compares the latest value against recent behavior for the same endpoint. Because what counts as “unusual” depends on the endpoint — a steady 100ms API route and a jittery search endpoint have very different normal swings — Scout measures each deviation relative to the endpoint’s own typical variability, not against a fixed threshold.

A single sensitivity setting controls how big a deviation has to be before Scout flags it. Higher sensitivity catches smaller deviations but produces more events; lower sensitivity catches only large, obvious spikes. A live preview on the endpoint chart shows exactly which past events each sensitivity level would have caught, so tuning isn’t guesswork.

Seasonality

Most apps have natural performance rhythms — a checkout endpoint that’s busier every weekday at 9am, a reporting job that runs heavy every night at midnight. Scout tries to account for these recurring patterns so you don’t get paged every time a known busy period kicks in.

When Scout sees a spike in the current minute, it cross-checks against two seasonal windows for the same endpoint:

• Same time, 24 hours ago — catches daily patterns (nightly cron jobs, end-of-day batch processing, daily traffic peaks).
• Same time, one week ago — catches weekly patterns (Monday morning traffic, weekend lulls, weekly billing runs).

If either window shows that the current value would have looked normal at that point in the past, the spike is treated as part of the endpoint’s usual rhythm and suppressed. Only deviations that are unusual for this time of day and this day of the week are flagged as anomalies.

This is automatic — there’s nothing to configure. New endpoints with less than a week of history will get a bit more noise during their first week as the seasonal model warms up; after that, recurring patterns are handled silently.

Enabling anomaly monitoring

You enable monitoring per endpoint or per background job from the Monitoring dropdown on its detail page.

1. Navigate to an endpoint or background worker
2. Click the Monitoring dropdown at the top of the page
3. Toggle Anomaly Monitoring on

When you enable monitoring, Scout immediately analyzes the past week of the endpoint’s history (you’ll see an “Analyzing recent history…" indicator while this runs). Any anomalies found in that week appear on the chart right away as retrospective events, so you can see what the monitor would have caught — and tune its sensitivity — without waiting for the next incident. These events will not be visible in insight history, and are intended mainly for tuning the sensitivty.

Going forward, Scout scans the endpoint every minute. Live detection needs roughly 30 minutes of traffic to establish a stable baseline before it flags new anomalies.

Disabling

Toggle Anomaly Monitoring off from the same dropdown, or remove the monitor from your organization-wide settings page (see below). Disabling stops new detection and hides the monitor’s chart markers; past anomaly events remain in your insights history.

Where to see anomalies

Once enabled, anomalies show up in several places:

Insights tab — A new Anomalies pill appears in the insights area of your app overview. Each item shows the metric, the endpoint or job, how far the value drifted from baseline (e.g. “3.5× above normal”), severity, and duration. Click through to the relevant endpoint with the time picker preset to the anomaly’s window.

Overview chart — On your app’s Overview page, a toggle above the main chart (alongside the alert and deploy marker toggles) overlays anomaly markers from all of your monitored endpoints and jobs on one chart.

Chart markers — On the endpoint or worker detail page, the same toggle will show anomalies as shaded bands on the performance chart. Red bands indicate values above baseline; blue bands indicate values below baseline. Hover any band for the underlying values. Anomalies found by the initial history analysis appear in a hatched, muted style and are labeled “Detected retrospectively” to distinguish them from events caught live.

Insights History — Past anomalies appear in the insights history view (alongside N+1, memory bloat, and slow query insights), so you can scan recent performance issues at a glance.

Notifications — Notifications are sent through your existing notification channels (see below).

Configuring sensitivity

Sensitivity controls how aggressive detection is. You can configure it at two levels:

Organization default — Sets the sensitivity for all monitors that don’t have a per-monitor override. Configure under Settings → Anomaly Monitoring.

Per-monitor override — On any monitored endpoint or background job, the Monitoring dropdown shows the monitor’s current sensitivity and an Adjust sensitivity button. Adjustments made here override the org default for that monitor only — useful when one endpoint is noisier or quieter than your typical traffic.

Adjusting sensitivity with the live preview

Clicking Adjust sensitivity opens a slider above the performance chart. The slider runs continuously from Low to High sensitivity, with three labeled presets along the way:

As you drag the slider, the anomaly markers on the chart filter live: only events big enough to fire at the candidate level stay visible, and a readout shows how many anomalies that level would have produced over the visible window. Because enabling a monitor backfills a week of history, you get a meaningful preview even on a brand-new monitor. For the clearest picture, set the chart’s time range to a full week — that’s how much history the backfill covers.

Nothing is saved until you click Apply; Cancel (or closing the dropdown) reverts the slider and the chart. You can stop on a preset or anywhere in between — in-between values are saved as Custom.

Two things to know about the preview:

• The preview shows what would have been detected in the past. Going forward, Scout only records events at your applied sensitivity — so apply the level you actually want before walking away.
• Lowering sensitivity hides smaller historical events rather than deleting them; raising it again reveals them.

Minimum duration (Advanced)

The preview bar has an Advanced section holding a second, optional control: Minimum duration. By default, a monitor opens an anomaly event — and sends its notification — on the very first anomalous minute. Raising the minimum duration makes Scout wait until the metric has stayed anomalous for that many consecutive minutes before opening an event. This filters out brief, self-correcting blips you don’t want to be paged for, at the cost of a slightly later alert on real incidents.

Open Advanced, then use the −/+ stepper (or type a value) to set the duration, from 1 to 60 minutes. Like sensitivity, this overrides the org default for that monitor only, and it’s saved together with the sensitivity threshold when you click Apply.

Raising the duration also filters the chart markers and the live count, so you can see roughly how many past events would have lasted long enough to fire. Unlike the sensitivity preview, the duration preview is approximate and tends to over-count — a note appears in the bar whenever the duration is above 1 minute. The value you save still applies exactly to future detection.

You can also set an org-wide default minimum duration under Settings → Anomaly Monitoring, which applies to any monitor without a per-monitor override.

Receiving notifications

Anomaly notifications use the same notification channels and groups that power Scout’s alerting — Slack, PagerDuty, email, and the rest. If you already have a notification group set up for alerts, you can reuse it for anomalies.

Pick a default notification group for anomalies under Settings → Anomaly Monitoring. All monitors send through this group. If no group is selected, anomalies are silent — visible in the dashboard, but no notifications. The Monitoring dropdown on each endpoint shows whether notifications are configured, with a link to set them up.

Notifications fire only when a new anomaly opens live. Scout does not re-notify while an anomaly is still active, does not currently send a “resolved” notification when it closes, and never notifies for retrospective events found by the initial history analysis.

What monitors best

Any endpoint or job you care about is worth monitoring. A few traffic shapes play to the detector’s strengths:

Throughput — Endpoints with at least 1 request per minute sustained will get a clean baseline. Below ~10 requests per hour, the trailing window is too sparse for the detector to flag anything meaningful, and you’ll get silence rather than useful signal. Very low-traffic endpoints are better handled by traditional alerting on absolute thresholds.

Consistency — Endpoints whose normal performance is reasonably stable produce the best signal. Endpoints with wild minute-to-minute swings (e.g. a search endpoint where some queries take 50ms and others 5s) will mostly look “always anomalous” to a statistical detector. Use response time alerting instead.

Latency matters — Anomaly monitoring currently covers response time and 95th percentile response time for endpoints, and response time for background jobs. It does not currently cover throughput or error rate. For throughput dips or error spikes, use traditional alerting.

Speed floor — Scout suppresses spikes that stay below user-perceptible latency, so very fast endpoints won’t generate events for tiny absolute changes (a 50ms → 90ms blip isn’t flagged even though it nearly doubled). If an endpoint averages well under ~200ms and you care about small absolute regressions, a threshold alert is the better tool.

You can add or remove monitors at any time.

Anomaly lifecycle

Open — When Scout detects the first anomalous data point that survives the seasonal check, it opens an anomaly event and (if a notification group is configured) sends a notification.

Active — While the anomaly persists, the event stays “active” in your dashboard. The event records its peak — the worst minute seen so far — so tooltips and insights describe the spike at its most severe, not just its latest value. No further notifications are sent during this period.

Closed — When the metric returns to its normal range for 3 consecutive minutes, the event closes. If the endpoint goes silent for 10+ minutes (no incoming traffic), Scout closes any open events for it automatically.

You can review closed anomalies in the insights history view.

Tips

• You can tune right away. Right after enabling a monitor, open Adjust sensitivity and drag the slider across the backfilled week of history — pick the level that would have caught the events you care about without flagging noise. You can come back and retune anytime as real events accumulate.
• Pair with alerting. Anomaly monitoring catches unusual behavior; threshold alerts catch unacceptable behavior. They complement each other.
• Use per-monitor overrides for noisy endpoints. You don’t need to lower your org default to accommodate a single chatty endpoint — override that monitor specifically.