Log Management

Scout’s Log Management feature allows you to monitor, search, and analyze your Python application logs directly within the Scout UI. By integrating with our existing APM agent, we provide enhanced context and filtering capabilities.

Installation

AInstall the Scout Python logging package:

pip install scout-apm-logging

BSet SCOUT_LOGS_INGEST_KEY in your existing configuration or via an environment variable. This key is provided in the Logs page of the enabled App.

CAdd the Scout logging handler to your Python logging configuration. Here's an example using dictConfig:

import os
from logging.config import dictConfig
from scout_apm_logging import ScoutOtelHandler

LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "verbose": {
            "format": "%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s"
        },
        "simple": {"format": "%(levelname)s %(message)s"},
    },
    "handlers": {
        "scout": {
            "level": "DEBUG",
            "class": "scout_apm_logging.ScoutOtelHandler",
            "service_name": "your-python-app",
        },
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "simple",
        },
    },
    "loggers": {
        "": {  # Root logger
            "handlers": ["console", "scout"],
            "level": "DEBUG",
        },
    },
}

# Apply the logging configuration
dictConfig(LOGGING_CONFIG)

This configuration attaches the ScoutOtelHandler to the root logger with the DEBUG log level, which will send all logs from internal libraries to Scout. Python logging is highly configurable, and we provide some common configurations below to get you started.

Common Configurations

By default, attaching the ScoutOtelHandler to the root logger with the DEBUG level will send all logs from internal libraries to Scout. While useful for comprehensive monitoring, this can result in a very large volume of logs, and likely more noise than signal.

Since the Python Logging agent provides a logging Handler, the source and severity of logs sent to Scout is configured via Python logging configuration.

There are many other possible logging configurations, but we hope these examples provide a useful starting point for some common needs.

Just Your App

This is a logging configuration which only attaches the “scout” handler to a custom logger: your_app

from logging.config import dictConfig
from scout_apm_logging import ScoutOtelHandler

LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
        },
    },
    "handlers": {
        "scout": {
            "level": "DEBUG",
            "class": "scout_apm_logging.ScoutOtelHandler",
            "service_name": "your-service-name",
        },
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "standard",
        },
    },
    "loggers": {
        "your_app": {
            "handlers": ["scout"],
            "level": "DEBUG",
            "propagate": True  # True by default
        },
    },
    "root": {
        "level": "WARNING",
        "handlers": ["console"],
    },
}

dictConfig(LOGGING_CONFIG)

Key Points

1. The root logger is set to WARNING level and only uses the console handler, effectively excluding other libraries' logs from Scout.
2. Propagation means that all logs will also be handled by the root handlers, so in this case: output to the console.
3. Adding the scout handler to the root would mean that WARNING or ERROR logs from all libraries would send to Scout. In this case, you could add the console handler to your_app and set propagate to False to not double-report.

Excluding Specific Libraries

Maybe you’re okay with getting most logs from various libraries, but want to reduce noise from especially verbose libraries.

### Initial configuration the same as above
...
    "loggers": {
        "your_app": {
            "handlers": ["console", "scout"],
            "level": "DEBUG",
        },
        "urllib3": {
            "level": "WARNING",
        },
        "boto3": {
            "level": "WARNING",
        },
    },
    "root": {
        "level": "INFO",
        "handlers": ["console", "scout"],
    },
}

Key Points:

1. Explicitly configuring loggers for known libraries (like urllib3 and boto3 in the example) gives you fine-grained control over their log levels.
2. Libraries that are not explicitly configured will inherit from the root logger, which is set to INFO level in this example.

Your application’s logger is set to DEBUG level, ensuring you capture all necessary details from your own code.

This configuration allows you to:

• capture detailed logs from your application
• reduce noise from verbose libraries by setting their levels higher
• catch logs from unconfigured libraries at the info level

How It Works

For Python applications, we provide a custom logging handler which wraps the OpenTelemetry Python SDK. This handler integrates with your existing logging setup, and sends logs directly to Scout without requiring the Otel Collector to be installed separately.

The logs will be available in the Scout UI for you to search and filter.

Data Retention

Scout retains your log data for 14 days. If you require longer-term storage, please let us know as we would like to create options for our customers, but at this point you will need to also send them to an alternative location.