June 14, 2016
"An ounce of prevention is worth a pound of cure." • Benjamin Franklin
Application performance problems can be annoying. With luck, you'll spend an hour or two resolving the problem and get back to your real job: building things.
But what happens when the issues start piling up? What happens when "poor performance" becomes the norm?
When looking at performance issues in aggregate and cross-application, a deeper root cause can appear. This can be tough for a team to identify and even tougher to act upon.
A common story: A business is thriving. The dev team gets away with bolting-on features and ignoring technical debt for a few years before the app slows to a grind at a brick wall of complexity. Performance at that brick wall is a constant struggle. Day-to-day development becomes difficult and convoluted.
This situation is so common that I've begun to promote certain behaviors as being prerequisites to maintaining a healthy and performant Rails app. Initially, these were technical behaviors like "keep Rails up to date" and "always watch your logs to see what SQL is firing." But these days, it's the developer relationship to the codebase that interests me most.
I call these prerequisites for performance because I firmly believe there is a causal relationship. When I see teams work with these attitudes, many performance problems are averted and handled early. Brick walls don't happen. It is far less costly for the business than the alternative: untangling and rebuilding after brick walls are hit, dealing with suffering customers, having development slow to a crawl...
This concept is not new or surprising. Think about the kitchen in your house. If you used it daily but didn't clean it for a year, it would not only be gross, it would become unusable. Imagine the work involved getting it back to a clean state...
Maintaining the kitchen on a daily basis is much less work and ensures it remains
I know how to do this better.
I can make this clearer.
This doesn't belong here.
This is unacceptable.
When these thoughts go through your head, listen to them and believe them. This is critical, especially on larger teams (5-50+ developers).
If a chunk of code isn't clear to you, chances are it's not clear to others on your team.
If you work through logic and discover something is ugly or in the wrong place, assume it's holding others back and fix it.
Assume you are right. Assume that you are smarter than the code (not your fellow devs!). If the code is complicated or confusing, it's not because you are slow or dumb; it's because the code could-no-should be better.
Rip the bad code out. Step all over it. As soon as parts of the codebase are considered precious and tiptoed around, it's a liability.
Paraphrasing Yoda: Fear is the path to the dark side. Fear leads to suffering. Suffering, in our case, is a messy and underperforming codebase.
Three fears I see regularly:
These are all the same thing: "We don't want to deal with this mess."
Often the most underperforming parts of applications are the hairy parts that haven't been touched in years, due to fear. Ironically, these are often old "performance optimizations" that weren't maintained.
You've seen this before. Most apps have "old school scary" code that devs on the team know about and tiptoe around. Maybe the dev who wrote the code left the company. Maybe it's critical but really hard to understand. Or there are no tests. Or the names of things are really bad.
Hotspots cost serious time and money as developers repeatedly butt up against (but don't resolve) the issue. We are afraid to dig in, so we work around instead. Complexity starts dripping "upstream" to the rest of the app. Before we know it, we are creating a SQL hairball or an esoteric hack...continuing the cycle of debt.
My rule of thumb: The more LOC considered scary or tiptoed around, the more likely the debt will need to be paid off down the line reactively as performance crumbles.
As for "fear of hard work": this fear dissipates as the team culture shifts and debt is paid back. Optimism kicks in. Hotspots are removed one by one. Devs discover that debt is easiest to pay off inline. Features become easier to build and modify again.
Ownership can be a problem, especially if there's more than a few devs working on an app.
Adding a new feature right next to some scary code? Improve the scary first. Maybe add a few tests.
Adding a new instance variable to a 20 line controller method? Break it apart.
In the hairiest of cases or the
This follows from the ownership issue. This is where I make references to the Broken Windows theory. Little things matter a lot. I define "better" condition as:
Intuitive Method and Variable Names.
strip_whitespace It requires
Never Copy and Paste. Think.
Imagine a 10 line scope
Before you commit, do some simple due diligence and assess if the logic can be reduced. Perhaps 6 of the lines are doing something convoluted to work around a missing association. Is there a table missing? A column? Why is this so complicated?
Never Copy and Paste. Refactor.
Remove. Unused. Code. Please.
Related: Never comment out
Comment on the "why"
Hypothetical: You spent 2 hours figuring out what a complicated method does. Fantastic!
Let's say you removed some of the complexity, but some of the "scary" still
If you copied a 10 line scope over
It condemns a future-you or a future teammate to handle it. Unless it's a private app, or the app won't have a long
Worse: Interest on technical debt compounds. Let's say a handful of other devs copy and paste the same scope into additional models. Someone else decides the scope should have another if/else condition and handle pagination, pushing it to 15 lines long. Not only is shotgun surgery now required to modify this common behavior, but the individual instances will diverge, encouraging bugs and comprehension difficulty. Not to mention that we are up to 105 LOC across 7 files.
Assume someone will have to clean up eventually. As Martin Fowler says:
We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design.
The original design could have been a perfect fit at some point, but it's
Building apps in Rails should feel like working with
I frequently see developers doing "too much work." For example, spending days composing and testing complex SQL queries or manipulating Active Record results with ruby. Occasionally these things are unavoidable, but a lot of cases can be gracefully solved in more maintainable ways via denormalization, adding columns or missing Active Record relationships.
When performance issues happen frequently in a Rails app, I often hear mutterings about Rails itself. Is it slow? Is it too magic? Is Active Record evil?
I consider these thoughts to be distractions. There is nothing stopping someone from serving thousands of snappy requests per minute with Rails and Active Record. Others are doing it happily.
Hard work should be reserved for figuring out how to present your domain as a web experience (UI design) and how to best represent your domain in data (database design). Both of these things are largely framework agnostic and can be quite difficult tasks. When Rails feels hard, or Active Record feels hard, consider it a "code smell." The first assumption should be that there's something about your UI or
The worst thing I can accuse Rails of: It can encourage an unrealistic feature development pace. It's easy to underestimate maintenance work needed and to keep bolting on more and more features. It can be hard to stop and spend time on fundamentals, such as data relationships, the UI, managing dependencies, etc.
Ultimately, these attitudes boil down to awareness.
The more aware your team is about the importance of maintainability, baking-in performance, and accurate data relationships, the
If your team is currently suffering from performance trouble, fear not! Your application is simply telling you it needs love and attention.
If your app is already at a brick wall, it is possible to recover and adopt a preventative stance. The only surefire way to make things worse is to continue to build on a weak foundation. This gets quite expensive for the business and very demoralizing for the team.
Instead, wipe the kitchen counters. Put away the dishes. Sweep the floor. And know that in the