Ruby CI Efforts
Charles Oliver Nutter: It seems that so far having head be broken for many days is not generally a concern for the “most core” of core developers. But it makes it nearly impossible for others to get up and running if builds fail and tests don’t run green.
My experience matches Charles'. And quite frankly it surprised me.
The Ruby core itself has test cases grouped in two tiers. One is the “bootstrapping tests” which mostly pass, and at any point in time has a small number of identified “known bugs”. This is mostly sane, but it is clear that Ruby developers have not instituted running this test suite as a general practice before every commit. Failed tests at this tier can go unnoticed for days. To their credit, if you do report the issue, it typically is resolved quickly.
The second tier of tests expands coverage to most of the core libraries. Tanaka Akira has been running and publishing these tests since 2005. But as near as I can tell, nobody is tracking the results. My experience is that having a specific and small test case coupled with the knowledge that it passed on revision n of the repository and fails on revision n+1 makes for quick problem determination. This team appears to make no effort to capitalize on this.
Alexey Verkhovsky has been running the Rails regression tests on Ruby 1.9. When I checked at the time, it showed an unbroken string of failures. It now appears to be down.
I am taking a different approach. I’ve taken a few small packages and gotten their tests to pass (and in some cases, it is only a subset of their tests), and only after those tests to pass am I adding them to my regression test. Once they are added, I try to stay on top of them. I am notified via XMPP on each failure, and when a failure occurs, I have both the last passing and current failing builds of Ruby on my machine to contrast with. As I try to expand the suite, I’ve come up with blocking issues of my own, but like the core team is doing, I’m now simply documenting them and moving on. This list is now down to one issue.
It occurs to me that this approach is highly distributable, with different people caring for different packages. It wouldn’t take many such people to build up a credible set of libraries “certified” to work on Ruby 1.9.
Despite all of these problems, I have become even more convinced that Ruby 1.9’s successor will quickly replace Ruby 1.8. Speed improvements (my experiences here are consistent with others), encoding support, real tasking, and some syntax improvements will cause people to want to upgrade. And the fact that it generally isn’t difficult to build libraries that work on both 1.8 and 1.9 means that a smooth upgrade is possible.