January 11, 2011
The New York Times ran a popular Op-Ed on Happiness last year. The Cliff Notes version?
The daily activities most associated with happiness are sex, socializing after work and having dinner with others. The daily activity most injurious to happiness is commuting.
Many software developers are fortunate to have flexible work hours that reduce (or eliminate) commute times. But what about that those times when you’re writing code and can’t have sex, socialize, or meet friends for dinner?
You need to be able to troubleshoot data-specific issues in the comfort and serenity of your development environment.
So, you need an easy way to grab production data and bring it local. In the simplest case, this is just a rake task that 1) ssh’s into your production db and
mysqldumps your database; 2)
gzips it and
scps it locally; 3)
unzips and imports the dump into your development DB.
That approach doesn’t scale. As your product grows, the all-or-nothing dump won’t be enough. Your database will be too big to dump quickly, and it will take too long to transmit over the wire. This is not the way to developer happiness.
What you need is a way to export a slice of your application’s data. There is no way to generalize this; it’s totally application specific. For Scout, the most sensible partitioning approach is to download one account and all its associated data. The trick is to identify the associated data to be export as it cascades through your application. The basic recipe we use for this is:
SelectiveExporterclass, passing the account ids we wish to export.
SelectiveExporterdoes a sequence of
mysqldumpcalls, assembling the set of data related to the account to export.
scpthe result and import it locally.
The payoff is huge: we’re able to grab data needed to troubleshoot a problem in 15 seconds instead of 30 minutes. That puts a smile on our developer faces.