Solving the random Rails performance problem

Last week I uncovered a tricky performance problem on Scout I wanted to share.

In seemingly random fashion, when accessing my account home page, the render time was 5-6 seconds – way too slow. Other times, the render time would be far less than a second.

Here’s what I did to debug:

  1. Tailed my production log
  2. Opened up the mysql console
  3. Generated a slow request
  4. Ran the mysql SHOW PROCESSLIST command


Plain as day, I saw a query taking 30+ seconds. A simple mysql EXPLAIN on the query revealed a missing index. A new index fixed the issue. This query was outside the Rails application – a background job – so I never saw the query when optimizing the application.

Checkout the impact on the DB server load:

It’s a great lesson that you really need the complete picture of your infrastructure when something goes wrong – just looking at the Rails application often doesn’t cut it.

P.S. You’ll love what’s coming up with Scout – finding these problems gets a lot easier.