Removing deadweight – cleaning up our Rails app

Every man has their breaking point when it comes to deadweight code. Andre and I hit ours recently and decided to spend all of last week focusing soley on cleaning up Scout (a Rails app). Our goals:

• Faster tests – our tests took 8 minutes to complete. While it’s the perfect amount of time to catchup on Daily Show clips, it really tested our patience making application-wide changes.
• Removing deadweight – unused CSS rules, database tables + columns, views, and assets. It’s good having certainty that modifying code will change something in the application.

Here’s how we went about it:

Tests

I needed visibility into our test performance – enter the test_benchmark gem. test_benchmark prints timing information for each test and test suite.

Two patterns emerged:

Slow External HTTP Requests

A number of our tests hit our payment provider’s gateway. All of these tests were slow. I mocked these requests with FakeWeb.

Time Savings: 3.76 minutes

ActiveRecord Generation

Some of our tests setup a decent number of database records. This can be significantly slower than using fixtures. Where appropriate, I switched to fixtures.

There’s a balance though between fixtures and creating fresh database records via ActiveRecord#create. When creating fresh records, you’re guaranteed to see records that match the application code (validations + callbacks run, default values are applied, etc). This isn’t the case with fixtures. It’s also a pain to update fixtures when logic / columns / etc. change. I looked for isolated cases – like recreating basic, seldom-changed records many times when running a test suite.

Time Savings: 35.7 seconds

In total, our test suite now takes 3 minutes to run – a huge improvement from 8 minutes.

Cleanup Unused Assets

Over the years, unused stylesheets, images, and javascript files have accumulated in our /public directory. It’s time to clear out what isn’t being used.

First step: look at our production logs for assets that have actually been requested recently.

We looked over the last week of log files—long enough to feel confident that every asset in use will have been requested. Here’s the grep command we used on Apache’s access.log:

The output of this is a long list of GET commands in assets.txt with lots of duplicate lines.

Second step: Delete assets that aren’t on that list

Andre created a quick ruby program to process assets.txt. Making this was an iterative process:

• Fleshing out the known_used list—because rails concatenates JS and css files for caching, the original component files don’t show up as “requested files.” In practice, this amounted to looking for :cache => 'all' in the code base, and iterating until the list of files looked right
• Every sparkline has a unique image url, but they aren’t relevant to asset cleanup. We used ignore_pattern to filter these out.

When all looked good, we uncommented the line in find_unused_assets.rb to actually delete the unused assets.

Third step: sanity check, checkin, sanity check on staging, deploy!

Unused CSS Rules, Views, database tables and columns

These involved mostly manual work:

• For CSS, I identified suspicious rules and searched for them in our view files. I removed those I couldn’t find.
• For views, I walked through our view folders, identified suspicious files, and searched for their names within render calls. I removed those that weren’t being used.
• For database tables and columns, I first looked for tables without correspending ActiveRecord models, removing those that weren’t referenced. I sanity checked the remaining tables, removing any tables + models that we no longer used. Columns worked in much of the same fashion – scanning our tables for suspicious columns and searching for their usage within the app.

I did make one mistake removing database columns: I forgot to run our Sinatra test suite (Sinatra handles checkins from our agents) after removing the database columns. For performance, we have some queries that select specific columns vs SELECT *. One of the columns I removed was referenced by a query in Sinatra. Double-check the entry points to your app!

Automated Tools

There are some automated tools that can help with asset cleanup:

CSS – The deadweight gem

From the README: “Deadweight is a CSS coverage tool. Given a set of stylesheets and a set of URLs, it determines which selectors are actually used and reports which can be “safely” deleted.”

Unused Partials – discover-unused-partials

From the README: “To use this script, simply run it in your RAILS_ROOT. It will return a list of unmentioned partials. It supports detection of Haml and Erb (both .erb and .rhtml) templates.”

I decided to go the manual route – we had a lot of low hanging fruit that was easy to discover w/o a separate tool.

The result: a more streamlined Scout

Our week of work resulted in some awesome diff stats:

Sometimes the first step to adding new functionality is clearing out what’s not being used – working on Scout is certainly more fun after a week of cleanup.

doug+scoutapp@scripted.com