January 15, 2018
A transaction trace is a GPS system for web performance: it paints a rich picture of the flow of a web request through your code.
So, why doesn't everybody trace? I believe there are two reasons:
logger.info()for logging or
Enter OpenTracing, a vendor-neutral open standard for distributed tracing. OpenTracing loosens the chains on tracing instrumentation: if we trace our method calls via OpenTracing APIs, we can swap out our tracing vendors just like logging and metrics!
To date, there are a number of services capable of ingesting transaction traces generated via OpenTracing. In this quick tutorial, we'll capture transaction traces from Python Flask app, and view this with Jaeger, an open-source distributed tracing system.
We developed an OpenTracing-compatible Python agent at Scout. You can learn more about it here.
You may have noticed that both OpenTracing and Jaeger's tagline mention "distributed". Besides providing a vendor-neutral instrumentation API, OpenTracing also provides a standard way to propagate trace context across microservices. This is crucial for debugging issues if your app is composed of microservices. I like to think of it as distributed backtraces.
I'll go over distributed Python tracing in a later post: we'll focus on instrumenting a single app in this tutorial.
This tutorial will use the following parts:
jaeger-client3.7.1 - the current version - does not yet support Python 3). I'm using Flask version 0.12.2.
The Jaeger All-in-one Docker image is designed for quick local testing. It launches the Jaeger UI, collector, query, and agent, with an in memory storage component. The
jaeger-client module installed in our Flask app sends traces to the collector on this running Docker container. Traces are then viewable in the web UI, accessible via
You can install and run Jaeger All-in-one via the following command (assuming Docker is installed on your machine):
docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp \ -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one:latest
Verify that the Web UI is accessible at
http://localhost:16686. Now that Jaeger is running, let's setup our Flask app.
Running the app:
FLASK_APP=opentracing_flask_step1.py flask run
The app's output:
$ curl http://localhost:5000 OpenTracing Pull Requests: Implement ScopeManager for in-process propagation, Implement global tracer support
Send a web request to
http://localhost:5000. Wait a moment, then navigate to
http://localhost:16686 and click the "Find Traces" button. You should see your first transaction trace!
Flask-Opentracing automatically traces web requests to our Flask app. However, it doesn't instrument method calls that occur within a web request. In our app, it'd be nice to instrument the GitHub API call and JSON parsing as these are two distinct areas of logic. To do this, we can wrap a span around each piece of code we'd like to instrument:
Note how I've added tags. This provides additional context for the span. Jaeger lets us search on these tags as well.
Send a web request to
http://localhost:5000, wait a moment, then refresh
http://localhost:16686. Inspect your most recent, richer trace:
With a couple of pieces, we've added vender-neutral tracing to a Python Flask app via OpenTracing. We can view these traces with Jaeger, an open-source distributed tracing system. If we'd like to use a tracing system other than Jeager at some point, we can swap out the tracer by changing a single line of code in our Python app.
OpenTracing not only defines a consistent API for tracing method calls. It also defines a consistent API for propagating transaction traces across requests handled by microservices. Soon I'll show how to trace a request across a Django and Flask app.