Rails Performance and the root of all evil
Donald Knuth wrote an often quoted paper in the 70s which
Premature optimization is the root of all evil.
In my line of work, it is sometimes invoked as a sort of apology; an excuse for why more time wasn't spent on performance: "This sucks, but at least we didn't.....prematurely optimize!"
My job is to fix performance issues and help other developers write well-performing code, so I'm not a huge fan of this quote. We shouldn't let an out of context quote guide our development strategy, no matter the source!
When things bother me, I like to stop to take a closer look. Here's the full quote:
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
I translate this to: "When optimizing, don't get distracted by things that don't matter."
I would never translate this to: "Hold off on optimizing things until it really hurts."
Identify What Matters
Some developers see performance work like they see taxes. It's the annoying obligation that comes after
Other devs stay away from performance work, remaining blissfully ignorant. They assume it is difficult, mysterious and elusive.
But what do we mean by performance work? I break it into 2 types:
-
"Baking in" performance with best practices and experience. In Rails, this would be proactively using pagination, having your
db indexes in order, having a good idea what SQL Active Record is firing,using .includes
where necessary, keeping track of where caching is going to be useful, etc. Not baking in these best practices along the way is akin to taking a mortgage out on your app. You are instantly creating debt and your options will be to "pay it off later" or bankruptcy. -
Measuring, identifying and handling performance issues. This is what I call "the problem of figuring out what the problem is." This is the work that can feel difficult, cumbersome and mysterious.
It's much easier to build performance into an app than it is to hunt down random performance problems as they bubble up. Baking in optimization is not what Knuth means by "premature." Let's see a bit more context around the Knuth quote:
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.
The context here is how to approach optimization, not whether to optimize or not. I would summarize his advice as:
- Don't be complacent (by avoiding optimization altogether)
- Once optimizing, it's easy to spend hours optimizing things that don't matter (this is evil)
- Instead, immediately identify the critical 3% that requires optimization
Measure Twice, Cut Once
Knuth is on a roll, so let's give him a bit more page time:
It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail
Got that? Knuth wants us to measure and discover where the problems are, not guess at the issues and dive right into trying to fix them.
I'm a huge fan of using my gut to guide
This is another reason why performance issue work might sound "hard." Endless hours spent down dead ends chasing "leads", fiddling around with things, coding and coding and coding...
Here are other things I see hours wasted on that I recommend skipping:
- Calling a meeting to discuss where the slowdowns might be (unless someone is bringing data/answers)
- Prototyping an abstraction to bypass a performance problem (or to preempt a future one)
- Blaming the framework or getting sidetracked by the framework's inherent compromises
- Assuming the scope of the problem is big or small before identifying where the problem is
Here is what I find to be the quickest path towards issue resolution:
- Go as big picture as possible and start collecting evidence
- Look at existing measurements from APM services like Scout, New Relic, Skylight
- You can also work locally, using logs or other tools
- Keep narrowing down the problem until you have one line or area at fault
If you don't have enough measurement around the problem, it's easy to make more. For example, once you know which action you care about, add rack-mini-profiler to your Gemfile, hit a few pages, and immediately get more detail.
If more than an hour or two has gone by and you have no idea where the problem is, don't despair. You can always try
Evil Optimizations
I’ve seen many underperforming Rails apps. In most cases, I walk
The favorite part of my job is when developers I work with agree and code more defensively, catching performance issues before they are introduced into the codebase. However, along with this increased attention, a new risk emerges - this is what Knuth is actually warning us about. The risk of eagerly optimizing "all the things" without measurement.
Here's a list of common "optimizations" I regularly see Rails devs tempted to chase down:
- Spending days learning about and profiling ruby memory leaks (Instead of measuring app code and rewriting some poorly crafted Active Record)
- Figuring out how to display 5000 rows of something on a page — it should be possible! (It is, but it will always be "lipstick on
pig ") - Evaluating whether an alternative database is necessary to get decent performance on what is a straightforward CRUD app (You can run, but you can't hide!)
- Building abstractions to work around performance problems (A great way to compound vs. reduce tech debt)
- Being distracted by which ruby object type might be allocating more memory (Spoiler Alert: This probably isn't the reason why the Rails processes are over 800MB...)
- Obsessing over rewriting parts of the app in Rust or C (Would a simple in-place refactor get us 95% of the way there? Or is this a number-crunching part of the app that warrants the big guns?)
Disclaimer: You may like these challenges! By all means, geek out, measure and explore. But most modern web apps do not require these types of optimizations. Default to spending time evaluating database relationships and Active Record usage, figuring out if the UI is sane, or measuring a custom abstraction that someone in the company built on top of Rails 3 years ago.
Non-Evil Optimizations
If your app has customers and is growing, and you don't proactively spend time optimizing popular features, heads up: it's likely you will eventually get complaints, downtime, and potentially a loss of business as your dataset and usage
- Ensuring that a popular action stays under 300ms for 95% of users
- Implementing caching for a page that is hit often and doesn't change regularly
- Increasing RAM as your database grows in size
- Ensuring that Rails instances stay at 50-70% capacity, so you have headroom for a traffic spike
- When expanding or adding features, evaluating the performance of the existing feature to see if it needs cleanup/refactoring first
- Monitoring your rails app regularly and fixing low hanging fruit like n+1s, lack of pagination, etc.
Nothing But the Knuth
My advice is to take a reasonable amount of time to bake in performance as you build. When trouble arises, figure out what the critical 3% is via measurement. Do this before allowing yourself to worry about esoteric topics or starting to shuffle code around