Finding and fixing N+1 queries in Django apps

The Django ORM makes it easy to fetch data, but there's a downside: it's easy to write inefficient queries as the number of records in your database grows.

One area where the ease of writing queries can bite you is with N+1 queries (Rails N+1 queries can kill performance too). Expensive N+1 queries go undiscovered in small development databases. Finding expensive N+1 queries is an area where Scout is particularly helpful.

Lets say you deploy a new Django view that renders a list of invoices and their associated retail store:

With a bare-bones development database, this performs just fine. However, when you deploy to production, performance will suffer significantly if a customer has many invoices.

When you add Scout to your Django app, Scout automatically identifies N+1s and calls these out on your app performance dashboard:

These insights are filtered to just N+1s consuming a significant amount of time, reducing noise. Also, since this is the first time an N+1 was detected for this view, the insight has a "NEW" indicator.

Clicking on an insight reveals a transaction trace for the request. Scout identifies the N+1 in the trace and provides a backtrace to the source of the query:

That's a lot of queries - 1000 separate calls to be exact!

In this case, the fix is straight-forward. Use select_related:

After deploying the fix, Scout continues to watch your new endpoint to see if an expensive N+1 re-emerges. When the N+1 hasn't appeared over a 24 hour period, the insight will disappear.

TL;DR

Knowing which N+1s are causing production problems is more efficient than optimizing every Django SQL query locally. When you add scout_apm to your Django app, you're able to see just the expensive queries and backtraces to their source, avoiding time-consuming pre-optimization work.

Sign up for Scout to get started.