From Ruby to Go: a rewrite for the future

Doug Breaker  on September 25, 2014

go

During a team camp among the lofty peaks of Breckenridge, Colorado, we talked a lot about the future of Scout and monitoring in general. Big mountains and nature have a way of doing that.

One thing that was getting our nerd juices flowing: Go.

At Monitorima in May, it was clear that Go was becoming the language of choice for performant yet fun-to-develop daemons.

After our morning hike fueled us with crip mountain air, we said: why not build a light Scout daemon in Go? As in, right this afternoon?

What followed was one of those programming sessions where everything flowed: by 11pm that night, we were doing things we didn't think were possible. We knew we'd be rewriting our agent in Go.

Our three takeaways below.

1. If you can build it, you can distribute it

coke

Our agent is currently written in Ruby. While most folks already have Ruby installed on their servers, how its installed is a different question. There's a standard Ruby install w/a system Ruby and system gems, Ruby Version Manager (RVM), Ruby Version Manager + Bundler, rbenv, and approximately 18 more ways your Ruby plus gems are installed.

Wouldn't it be great if you could release something and you knew it would just run?

Welcome to Go. Produce binaries for foreign platforms right from your desktop.

Lets start with a simple hello_world.go:

package main

import(
    "fmt"
)

func main() {
    fmt.Println("Hello World!")
}

I'm running OSX. I'll build for Windows:

$GOOS=windows GOARCH=amd64 go install

Looking in my bin directory, I see:

$ls bin/windows_amd64/
hello_world.exe

I'll throw this on Dropbox and then run it on a Windows computer:

windows go

Now, there are a few gotchas, but it's significantly more efficient than using a tool like Omnibus to build a self-contained Ruby executable.

2. Sacrifice some aesthetics for performance

spoiler

I've never had more fun writing code than using Ruby. It's an easy, non-statically-typed syntax to pick up. Go isn't bad - certainly better than Java. Here's a simple piece of code that iterates over some URLs and prints them.

In Ruby:

urls = ["http://www.cnn.com","http://espn.go.com/","http://grantland.com","http://www.newyorker.com"]

urls.each do |url|
    puts "URL: #{url}"
end

In Go:

package main

import (
    "fmt"
)

func main() {
    urls := []string{"http://www.cnn.com","http://espn.go.com/","http://grantland.com","http://www.newyorker.com"}

    for _, url := range urls {
        fmt.Printf("URL: %s\n",url)
    }
}

The result of both:

URL: http://cnn.com
URL: http://espn.com
URL: http://grantland.com
URL: http://newyorker.com

My wife, a 5th-grade teacher that doesn't write code, could guess what the Ruby snippt does. The Go snippet is more cryptic, but for a developer, isn't terrible to grok. However, I certainly wouldn't switch to Go because I find the code easier to read than Ruby.

Some loss in readability though is a good compromise for much better performance.

Our existing Ruby agent is light enough on resources (cpu usage around 0.2%, memory usage around 30 MB), but there's a ceiling on providing more intensive, but very valuable monitoring data with Ruby. A switch to Go is about expanding possibilities.

Ruby is significantly slower and more memory-hungry than Go (anywhere from 10x - 100x). Go is slower than Java, but its getting better, is still young, and I find easier to read.

Trading some readability for significant performance and resource usage improvements is worth it for a monitoring agent.

3. Concurrency so easy, you might not know you're doing it

Lets say you need to poll a number of URLs and ensure the URLs are responding correctly. I'll walk through the URLs one-by-one and check their status.

Here's the Ruby version:

require 'net/http'
require 'uri'

urls = ["http://www.cnn.com","http://espn.go.com/","http://grantland.com","http://www.newyorker.com/"]

urls.each do |url|
    response = Net::HTTP.get_response(URI.parse(url))
    puts "#{url}: #{response.code} #{response.message}"
end

In Go:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    urls := []string{"http://www.cnn.com","http://espn.go.com/","http://grantland.com","http://www.newyorker.com/"}

    for _, url := range urls {
        response, err := http.Get(url)
        if err != nil {
            // error handling
        }
        fmt.Printf("%s: %s\n",url,response.Status)
    }
}

The result:

http://www.cnn.com: 200 OK
http://espn.go.com/: 200 OK
http://grantland.com: 200 OK
http://www.newyorker.com/: 200 OK

It takes about 1.2 seconds for Ruby to check these URLs and about 0.5 seconds for Go to do the same. That's a nice improvement, but could we make this even faster?

Bill O'Reilly hacks CNN

cnn

What if Bill O'Reilly and Fox News hijack the CNN website and performance suffers? It will take a bit of time for Anderson Cooper to jump on it. While cnn.com is running slow, the remaining URLs will need to wait for the cnn.com url check to complete. This stinks - the CPU is just twiddling its thumbs during this time, waiting on the network. It could be doing more work.

We could be running these URL checks concurrently.

Go makes starting concurrent tasks easy with Goroutines:

package main

import (
    "fmt"
    "net/http"
    "sync"
)

func main() {
    urls := []string{"http://www.cnn.com","http://espn.go.com/","http://grantland.com","http://www.newyorker.com/"}
    var wg sync.WaitGroup
    wg.Add(len(urls))
    for _, url := range urls {
        go func(url string) {
            defer wg.Done()
            response, err := http.Get(url)
            if err != nil {
                // error handling
            }
            fmt.Printf("%s: %s\n",url,response.Status)
        }(url)
    }
    wg.Wait() // waits until the url checks complete
}

With Goroutines in place, I've seen total execution times around 0.16 seconds, a 7.5x improvement vs Ruby:

http://www.newyorker.com/: 200 OK
http://grantland.com: 200 OK
http://espn.go.com/: 200 OK
http://www.cnn.com: 200 OK

real    0m0.156s
user    0m0.015s
sys 0m0.026s

But it's 2014. Surely you can do concurrency with Ruby.

It's true. You can. But there are a number of gotchas: it's the difference between starting a language with concurrency in mind versus adding it later.

Join our BETA

The Go agent is going to open up some exciting doors. If you live on the bleeding edge, shoot us an email and we'll add you to our BETA list.

Also, don't worry: we'll continue supporting our existing Ruby agent.