Rails 2.3.6, 2.3.7, 2.3.8, all in less than 48 hours. My tests played a role in getting rails back on course. Overall, I continue to believe that it is much more important for a project to be responsive than correct. Ideally, however, these problems would have been noted before shipping. At least two corrections make sense in response:
The problems fixed in 2.3.7 were real problems, just ones that don’t affect the readers of Agile Web Development with Rails, edition 3. Specifically the rails_xss plugin, which was introduced both to address a specific security issue and to aid in the transition to Rails 3, caused problems for HAML. More continuous testing of both this plugin and HAML are in order.
While Rails has an impressive number of tests, nothing replaces actual usage. My tests passed on 2.3.6, and indicated at least two problems of the problems that needed to be fixed in 2.3.7, and pass again on 2.3.8. We need more people to do similar things. And we need the Rails team to look for indications that these types of test pass before shipping. I’ll be discussing this at RailsConf in just over two weeks.
A slight correction (I’m the author of Haml): the bugs in Rails 2.3.6 were not limited to its interactions with Haml and other alternate templating engines, although there were bugs of that sort as well. The reason 2.3.7 was released so soon was that my investigation into the bugs affecting Haml turned up some serious security issues which affected at least all users of Rails' XSS protection.
In addition, I wouldn’t say that rails_xss caused the issues. The way the XSS-protection backport works is that most of the XSS-protection code is in Rails proper (which incidentally worked fine in Rails 2.3.5), and rails_xss just activates it. The bugs were caused by a modification of the XSS protection in Rails, and indeed affected Haml whether or not rails_xss was installed. It is true that the security bug that necessitated 2.3.7 only affected rails_xss users, though.
Nathan: thanks for the clarification. From my perspective, none of this obviates the need for more system testing prior to releases of Rails. I personally don’t believe that having the Rails team testing all possible permutations scales.
The discussion I would like to motivate: what can be done to identify potential breakages to both the core team and to library maintainers? Ideally this would be done closer to the time the change is made, but in any case before a release is made?
My experience is that if I communicate a problem to the core team within a few days of when the change was introduced, problems will be addressed within a few days. Often this feedback loop is much, much tighter than that: often I can report a problem within hours and the fix is made within hours.
I agree that more system tests would be good, but in this case finding at least the issues with 2.3.6 would have been more straightforward: if there were any tests for rails_xss, they would have revealed these issues. There were several layers of redundancy already in place that should have caught these issues. Wider testing would provide another layer to catch more issues faster, which is great, but I think it’s important to recognize that this change would have been caught by the standard methods of detecting errors had they been applied.