Some of our team just got back from two pretty intense weeks of conferencing. We had an amazing time speaking at the JAX, Railsberry and RailsConf and getting our hands on two of the Ruby Hero Awards. A couple of announcements were made live on stage and you might have heard a few rumors coming out of the conferences. So let me wrap up one of them for you.

I’m pretty excited about this one, as I’ve been working on it over the last weeks. This is the first feature to be sponsored by our impressing Love Campaign. While the promise of adding more languages actually came true, this was mainly done by excellent community contributions and the hard working Michael Klishin.

How We Use Pull Requests

When GitHub first announced Pull Requests 2.0, it wasn’t obvious right away how truly amazing this feature is for us developers.

It revolutionized the workflow by making the patches part of the discussion. This actually led to code reviews from other contributors while working on a feature, rather than once you think you’re done working on it.

Think of it as bringing agile development to Open Source contributions. Wikipedia even calls this new and social approach The GitHub revolution.

It really only had one downside: Merging still was as complicated as before. You have all the fancy review tools and still have to go into the terminal and type a couple of commands. This quickly became rather tedious.

The Merge Button

GitHub, once more, came to our rescue by adding the Merge Button.

By pressing a button on the website, one could easily merge Pull Requests without having to drop to the console.

This feature, combined with the Fork and Edit button, made contributing a no brainer.

Especially the round-trip time for documentation fixes, like typos, broken examples, etc. went down to sometimes just a few seconds.

Contributing a fix became as simple as two clicks on GitHub. Making contributions that easy lowers the barrier and thereby strengthens the Open Source ecosystem.

You probably can tell by now how much we love this feature.

There is a downside, though. The Merge Button is not really usable for anything but documentation.

If you click the Merge Button for anything touching code, you risk breaking upstream. Maybe this seems acceptable at first, after all your CI will tell you if you broke anything. You are using CI, right?

However, it not only renders your mainline pretty unstable, it also changes responsibility. All of a sudden you as a maintainer are responsible for fixing the issue, if only by reverting the commits.

So you’re back at merging locally and checking out if everything works. Maybe you even push on a feature branch first, just to trigger a CI run before merging.

Enter Travisbot

It would be really cool to just know if the Pull Request breaks anything, without all the hassle.

This is basically what we’ve implemented. We test every mergeable Pull Request and have our friendly Travisbot leave a comment in the Pull Request discussion.

That way you can now safely press the Merge Button. That is, if it doesn’t break anything, of course. And if it breaks, it’s not necessarily the maintainers’ (or worse, the users’) responsibility to deal with the issue.

Or imagine having a broken upstream and someone submits a Pull Request fixing it.

In contrast to most self-made solutions out there, we actually test the merged version, rather than just the fork or feature branch. Thus, we also take into account changes made upstream after the repository has been forked.

We will also re-run the tests whenever new commits are added to Pull Requests (yes, this works fine with force pushes and rebases). Also, it pretty much works with anything that is mergeable, might it be a branch, tag, fork, etc. As long as you see the green Merge Button, we can test it.

Are we there yet?

This feature is still under development. The described functionality works just fine, but we have plans for future extensions.

For instance, we would like to leave a comment whenever master is updated and give you the option to automatically rerun the tests with the new merge commit.

I WANT THIS NOW!!

If you made it this far in the blog post, you probably want it for your repository, and you want it now. Good news! We started unrolling this feature.

As we are still improving it, and are not sure if we can stand the load of activating it for everyone right away, we are turning it on on a repo by repo basis.

Please note that we started unrolling for those who donated, so I was actually able to work on this. So, if you donated and want this, just let us know.

If you didn’t donate, but still want to use this feature right away, our Love Campaign is still running.

Making sure Travis always runs smoothly all the time has become one of our most important tasks over the recent weeks and months. As a distributed system, there are a lot of moving parts, and we’re putting more and more eyes on every single piece to make sure they do their job. Obviously there’s no need to keep this a secret, so let’s talk about some recent changes to Travis’ infrastructure.

Case in point. A recent deployment broke updating profiles temporarily. Here’s a pretty graph showing pretty much the exact point in time when we deployed and the spike in the number of errors.

You can see the initial spike, which represents a sudden increase in application errors. We rolled out a fix shortly after, so things quieted down somewhat. There’s still some noise though, still several 500 errors popping up in our logs. It took a while to fix those, but you can see it eventually ebbed off after another deployment.

This is a pretty powerful tool. We added more and more metrics on the moving parts in Travis. If you want to know why, you should read this blog post from the Flickr engineering team, and you should definitely watch Coda Hale’s talk Metrics, Metrics Everywhere!.

All these metrics tell us what’s going on inside Travis in real time, and they show us historical growth so we can spot patterns, capacity issues and unexpected spikes. Here’s a another pretty graph of all our internal notifications. These metrics are pretty new and were put in production last Friday, but it’s still something to enjoy (click to enlarge).

All these pretty graphs are courtesy of Librato Metrics, who were kind enough to offer sponsoring their service for Travis CI. As we add more and more metrics to Travis they’ll automatically show up in Librato and will be graphed in real time.

The fun doesn’t stop there. To make it possible that we can properly graph 404 and 500 errors, we added custom logging. Rails’ default logging is not very helpful for production systems, as it produces a lot of noise. Thanks to Rails 3 and the new notifications introduced with it, it’s now possible to override what Rails does by default and hook in your own logging. That’s exactly what we did, and the result is lograge, a little library that turns Rails’ logging events into a neat, single line per request. You can read all about it in Mathias’ blog post.

This helped streamline our logging, making it more and more useful along the way. All our logs are aggregated in Papertrail, who were also kind enough to offer sponsoring their services for Travis.

This allowed us to do something pretty nifty. Eric of Papertrail wrote a neat little metrics library for Ruby called Metriks, which we started using in Travis. It collects metrics and can dump them to either a log file or directly export to Librato Metrics or Graphite. It can also show selected metrics in the process status, so whenever you run ps ax, you can see the current number of requests per second, or pretty much whatever you fancy. You should go and read Eric’s blog post on the library.

The really neat bit is that Metriks can log the collected metrics to the log file. Papertrail can then look for patterns in the aggregated logs and react accordingly, in this case triggering a webhook where the metrics are aggregated and eventually stored in Librato Metrics. Did that just blow your mind?

Here’s how that looks coming into Papertrail (click to enlarge).

This little trick utilizes Papertrail’s filters and webhooks alerts, and a nifty little app that Eric built to accept the filtered output when a metric is found in the logs.

Why would we want to do that? Simply because it decouples the fact that we’re collecting from where the metrics need to be stored. It’s a fully asynchronous process, and they’ll simply show up in Librato eventually. We can also put a filter on anything in Papertrail and have occurrences in the log be directly graphed in Librato’s Metrics. It’s pretty awesome. Every bug fixed is a new opportunity to add more graphs and add more logging.

For good measure, here’s another graph showing the requests we’re receiving from GitHub. It includes the received, accepted and rejected pushes.

The beautiful bit is that we can take any metric and correlate it with others. That’s a pretty powerful tool to find problems in infrastructure.

This is far from all of it, but it’s a start. If you want to know more about why Mathias is so obsessed with metrics and monitoring, you should read his blog post about it. See you in the next installment about how we’re adding more mustache to Travis’ infrastructure.

Keeping Up With the Times

When we first started using virtual machines for Travis CI (around June 2011) we decided to use Ubuntu 10.04. This worked perfectly, but by the fall of 2011 10.04 started showing its age. Our users kept asking for more recent versions of certain tools and libraries which were challenging to provide without building and maintaining a myriad of Debian packages. So in November 2011 we migrated all VMs to Ubuntu 11.04 which solved the problem.

Now it is Spring 2012 and the time to move on to 11.10 is drawing close. We want to explain briefly how Travis CI will migrate to it, why we are doing it and what may change for your project.

Why Migrate?

With Ubuntu 11.10, we will be able to provide more up-to-date versions of tools and services in our CI environment, including:

• OpenJDK 7 alongside OpenJDK 6 (this is still a work in progress)
• PostgreSQL 9.1 + updated extensions
• CouchDB 1.1
• Firefox 11

In addition, we won’t have to use 3rd party apt repositories or compile certain tools from source, which greatly reduces our maintenance burden, for example:

• Haskell Platform
• Maven 3

Staying One Step Behind, Intentionally

Our users are mostly software developers and they tend to like to staying up-to-date with tools, services, libraries and so on. However, production environments are rarely on the bleeding edge. So for CI in general, and Travis CI in particular, it is important to maintain a balance: not too old, but not too new either. This is why Travis CI is intentionally about 6 months behind Ubuntu releases. It gives developers several months to catch up with recent changes, fix issues and push out new releases.

Important Changes in 11.10

GCC

One of the biggest changes with every distro release is GCC version changes. 11.10 ships with GCC 4.6. This version is known to be more strict than previous releases and some tools and libraries may need minor changes when compiled with -Wall. Judging from the experience from one of our team members, most popular tools were updated in the beginning of this year.

OpenSSL

Another important change OpenSSL version. 11.10 ships with OpenSSL 1.0 by default, but even more importantly, disables SSLv2 support. SSLv2 is very old (released in 1995), there were four updates to the spec after it: SSL 3.0, TLS 1.0, TLS 1.1 and TLS 1.2. In addition to that, SSLv2 contains several known security flaws and all new distributions either already dropped support for it or will be dropping it very soon.

PHP 5.2

Because of the SSLv2 change explained above, we had to disable SSL support for PHP 5.2 and 5.3.2. PHP 5.2 is no longer maintained by the PHP Core Team and we were faced with a decision: either drop it completely or patch and maintain it to work with OpenSSL 1.0, or disable SSL support for it.

PHP 5.3.10 and 5.4.0 are maintained and work with OpenSSL 1.0 without modifications, and have OpenSSL support enabled, just like they have today.

The Road to 11.10

On April 6th, 2012 we plan to update all our VMs to use Ubuntu 11.10. We will provide the same set of tools, services, and libraries, but some of them will be newer than what we have today. Most maintained software should work without any changes on your part. If you experience any issues please drop by #travis on chat.freenode.net and we will do our best to help you.

You can also try a minimalistic 11.10 image using Virtual Box and sous chef or Vagrant with our 11.10 base box.

Happy testing!

The Travis CI Team