Ruby CI Efforts
Charles Oliver Nutter: It seems that so far having head be broken for many days is not generally a concern for the “most core” of core developers. But it makes it nearly impossible for others to get up and running if builds fail and tests don’t run green.
My experience matches Charles'. And quite frankly it surprised me.
The Ruby core itself has test cases grouped in two tiers. One is the “bootstrapping tests” which mostly pass, and at any point in time has a small number of identified “known bugs”. This is mostly sane, but it is clear that Ruby developers have not instituted running this test suite as a general practice before every commit. Failed tests at this tier can go unnoticed for days. To their credit, if you do report the issue, it typically is resolved quickly.
The second tier of tests expands coverage to most of the core libraries. Tanaka Akira has been running and publishing these tests since 2005. But as near as I can tell, nobody is tracking the results. My experience is that having a specific and small test case coupled with the knowledge that it passed on revision n of the repository and fails on revision n+1 makes for quick problem determination. This team appears to make no effort to capitalize on this.
Alexey Verkhovsky has been running the Rails regression tests on Ruby 1.9. When I checked at the time, it showed an unbroken string of failures. It now appears to be down.
I am taking a different approach. I’ve taken a few small packages and gotten their tests to pass (and in some cases, it is only a subset of their tests), and only after those tests to pass am I adding them to my regression test. Once they are added, I try to stay on top of them. I am notified via XMPP on each failure, and when a failure occurs, I have both the last passing and current failing builds of Ruby on my machine to contrast with. As I try to expand the suite, I’ve come up with blocking issues of my own, but like the core team is doing, I’m now simply documenting them and moving on. This list is now down to one issue.
It occurs to me that this approach is highly distributable, with different people caring for different packages. It wouldn’t take many such people to build up a credible set of libraries “certified” to work on Ruby 1.9.
Despite all of these problems, I have become even more convinced that Ruby 1.9’s successor will quickly replace Ruby 1.8. Speed improvements (my experiences here are consistent with others), encoding support, real tasking, and some syntax improvements will cause people to want to upgrade. And the fact that it generally isn’t difficult to build libraries that work on both 1.8 and 1.9 means that a smooth upgrade is possible.
Schwern,
The git distributed version control system supports binary bug searching, in the form of git-bisect.
Posted by Jon atTab Sweep—Ruby
I was doing a general Tech Tab Sweep and I noticed that I had a ton of Ruby-related stuff, so it gets its own fragment....Excerpt from ongoing at
CI done right
Dear Sam , Although what you describe comes quite close to CI , it isn’t. Unless you compile the code and run the test suite in a clean and neutral build environment, you can never be sure that the changes you’re about to commit really do integrate...Excerpt from AK's weblog at
CI done right
Dear Sam , Although what you describe comes quite close to CI , it isn’t. Unless you compile the code and run the test suite in a clean and neutral build environment, you can never be sure that the changes you’re about to commit really do integrate...Excerpt from AK's weblog at
The weekly bag– Feb 1
Enjoy these fine links: TestNG and Scala- it seems it may be high time to learn Scala! Observing Behavior- Igor has some compelling thoughts about “verifying observable behavior.” Ruby CI Efforts- interesting thoughts around community...Excerpt from The Disco Blog at
Sounds like you’re on the right track.
This sounds a lot like Andreas Koenig’s method for testing the development version of Perl which has been termed “the BBC” or Bleadperl Breaks CPAN. During the run up towards the 5.10.0 release he was doing them every day. You can see one of his summary reports here:
[link]
Essentially, Andreas runs a snapshot of the development perl against a battery of important, or simply traditionally cantankerous, Perl libraries and reports the interesting failures. This allows Perl to use CPAN as an informal test suite better approximating the real world. Because CPAN has a uniform structure automating the process is easy. It allowed 5.10.0 to be one of the most bug-free releases yet.
Andreas has all sorts of fascinating perl development tools. One is a binary bug search tool. If we find a bug he can write up a small test script and run it against various revisions of perl to find what change broke it. He employs a simple binary search algorithm (and a lot of spare disk space) to track it down.
Posted by Schwern at