A Six-Pronged Rails Performance Philosophy

"An ounce of prevention is worth a pound of cure." • Benjamin Franklin

Application performance problems can be annoying. With luck, you'll spend an hour or two resolving the problem and get back to your real job: building things.

But what happens when the issues start piling up? What happens when "poor performance" becomes the norm?

When looking at performance issues in aggregate and cross-application, a deeper root cause can appear. This can be tough for a team to identify and even tougher to act upon.

The Brick Wall

A common story: A business is thriving. The dev team gets away with bolting-on features and ignoring technical debt for a few years before the app slows to a grind at a brick wall of complexity. Performance at that brick wall is a constant struggle. Day-to-day development becomes difficult and convoluted.

This situation is so common that I've begun to promote certain behaviors as being prerequisites to maintaining a healthy and performant Rails app. Initially, these were technical behaviors like "keep Rails up to date" and "always watch your logs to see what SQL is firing." But these days, it's the developer relationship to the codebase that interests me most.

I call these prerequisites for performance because I firmly believe there is a causal relationship. When I see teams work with these attitudes, many performance problems are averted and handled early. Brick walls don't happen. It is far less costly for the business than the alternative: untangling and rebuilding after brick walls are hit, dealing with suffering customers, having development slow to a crawl...

This concept is not new or surprising. Think about the kitchen in your house. If you used it daily but didn't clean it for a year, it would not only be gross, it would become unusable. Imagine the work involved getting it back to a clean state...

Maintaining the kitchen on a daily basis is much less work and ensures it remains useable. A small spill is just a small spill.

1. Be Arrogant (With The Codebase)

I know how to do this better.
I can make this clearer.
This doesn't belong here.
This is unacceptable.

When these thoughts go through your head, listen to them and believe them. This is critical, especially on larger teams (5-50+ developers).

If a chunk of code isn't clear to you, chances are it's not clear to others on your team.

If you work through logic and discover something is ugly or in the wrong place, assume it's holding others back and fix it.

Assume you are right. Assume that you are smarter than the code (not your fellow devs!). If the code is complicated or confusing, it's not because you are slow or dumb; it's because the code could-no-should be better.

Rip the bad code out. Step all over it. As soon as parts of the codebase are considered precious and tiptoed around, it's a liability.

2. Never Be Afraid

Paraphrasing Yoda: Fear is the path to the dark side. Fear leads to suffering. Suffering, in our case, is a messy and underperforming codebase.

Three fears I see regularly:

  1. Fear that the code is difficult to understand.
  2. Fear that touching code will break it.
  3. Fear of "hard work."

These are all the same thing: "We don't want to deal with this mess."

Often the most underperforming parts of applications are the hairy parts that haven't been touched in years, due to fear. Ironically, these are often old "performance optimizations" that weren't maintained.

You've seen this before. Most apps have "old school scary" code that devs on the team know about and tiptoe around. Maybe the dev who wrote the code left the company. Maybe it's critical but really hard to understand. Or there are no tests. Or the names of things are really bad.

Hotspots cost serious time and money as developers repeatedly butt up against (but don't resolve) the issue. We are afraid to dig in, so we work around instead. Complexity starts dripping "upstream" to the rest of the app. Before we know it, we are creating a SQL hairball or an esoteric hack...continuing the cycle of debt.

My rule of thumb: The more LOC considered scary or tiptoed around, the more likely the debt will need to be paid off down the line reactively as performance crumbles.

As for "fear of hard work": this fear dissipates as the team culture shifts and debt is paid back. Optimism kicks in. Hotspots are removed one by one. Devs discover that debt is easiest to pay off inline. Features become easier to build and modify again.

3. You Touch It, You Own It

Ownership can be a problem, especially if there's more than a few devs working on an app.

Adding a new feature right next to some scary code? Improve the scary first. Maybe add a few tests.

Adding a new instance variable to a 20 line controller method? Break it apart.

In the hairiest of cases or the rushiest of rushes: make a mental note that it needs to be refactored. Bring it up with your boss. Explain that it would be irresponsible to build further there without some refactoring. If you lack a test harness to do this efficiently, fantastic! There is a clear step 1.

4. Always Leave the Code In Better Condition Than You Found It

This follows from the ownership issue. This is where I make references to the Broken Windows theory. Little things matter a lot. I define "better" condition as:

Intuitive Method and Variable Names.

Variable names like @usr_pro_img are not as clear as @user_profile_image. The same with a local variable p compared with posts or a method process_string vs strip_whitespace It requires more collective brain to parse and maintain esoteric and ambiguous names. Good names remove future "scary" and saves everyone time.

Never Copy and Paste. Think.

Imagine a 10 line scope on post.rb — it looks hairy. You think you know what it does. You need something just like it on comment.rb. You could just copy it over and change a few variable names...

Before you commit, do some simple due diligence and assess if the logic can be reduced. Perhaps 6 of the lines are doing something convoluted to work around a missing association. Is there a table missing? A column? Why is this so complicated?

Never Copy and Paste. Refactor.

Do post.rb and comment.rb share a lot of other behavior? Is it appropriate to do a bigger refactor here? Would that result in clearer code?

Remove. Unused. Code. Please.

Related: Never comment out code and leave it in the file. It's not educational. It doesn't provide context. It's cruft. Remove it. This is what git history is for.

Comment on the "why"

Hypothetical: You spent 2 hours figuring out what a complicated method does. Fantastic!

Let's say you removed some of the complexity, but some of the "scary" still remains. It's always best to make the effort to explain "why" in a concise comment to help the next dev (or the future you) save time.

5. Handle It Now, Or It's Debt

If you copied a 10 line scope over from post.rb to comment.rb and changed a few names, I would call it technical debt.

It condemns a future-you or a future teammate to handle it. Unless it's a private app, or the app won't have a long life-expectancy, it's best to handle things like this now.

Worse: Interest on technical debt compounds. Let's say a handful of other devs copy and paste the same scope into additional models. Someone else decides the scope should have another if/else condition and handle pagination, pushing it to 15 lines long. Not only is shotgun surgery now required to modify this common behavior, but the individual instances will diverge, encouraging bugs and comprehension difficulty. Not to mention that we are up to 105 LOC across 7 files.

Assume someone will have to clean up eventually. As Martin Fowler says:

We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design.

The original design could have been a perfect fit at some point, but it's not longer appropriate. The app requires regular pruning and maintenance.

6. Treat Rails Like Legos

Building apps in Rails should feel like working with legos. Ideally, you are happily snapping the pieces together. If your team is constantly melting down plastic and fusing pieces together, it's a red flag.

I frequently see developers doing "too much work." For example, spending days composing and testing complex SQL queries or manipulating Active Record results with ruby. Occasionally these things are unavoidable, but a lot of cases can be gracefully solved in more maintainable ways via denormalization, adding columns or missing Active Record relationships.

When performance issues happen frequently in a Rails app, I often hear mutterings about Rails itself. Is it slow? Is it too magic? Is Active Record evil? Is vanilla Rails enough?

I consider these thoughts to be distractions. There is nothing stopping someone from serving thousands of snappy requests per minute with Rails and Active Record. Others are doing it happily.

Hard work should be reserved for figuring out how to present your domain as a web experience (UI design) and how to best represent your domain in data (database design). Both of these things are largely framework agnostic and can be quite difficult tasks. When Rails feels hard, or Active Record feels hard, consider it a "code smell." The first assumption should be that there's something about your UI or db schema that is unideal and the difficulty is spreading from there.

The worst thing I can accuse Rails of: It can encourage an unrealistic feature development pace. It's easy to underestimate maintenance work needed and to keep bolting on more and more features. It can be hard to stop and spend time on fundamentals, such as data relationships, the UI, managing dependencies, etc.

Awareness

Ultimately, these attitudes boil down to awareness.

The more aware your team is about the importance of maintainability, baking-in performance, and accurate data relationships, the more able they are to casually prevent performance issues and brick walls as they build.

If your team is currently suffering from performance trouble, fear not! Your application is simply telling you it needs love and attention.

If your app is already at a brick wall, it is possible to recover and adopt a preventative stance. The only surefire way to make things worse is to continue to build on a weak foundation. This gets quite expensive for the business and very demoralizing for the team.

Instead, wipe the kitchen counters. Put away the dishes. Sweep the floor. And know that in the end you'll have a fast, humming machine.